Google Associate-Cloud-Engineer Google Cloud Certified - Associate Cloud Engineer Exam Practice Test

Cloud Functions is Google Cloud’s event-driven serverless compute platform that lets you run your code locally or in the cloud without having to provision servers. Cloud Functions scales upor down, so you pay only for compute resources you use. Cloud Functions have excellent integration with Cloud Pub/Sub, lets you scale down to zero and is recommended by Google as the ideal serverless platform to use when dependent on Cloud Pub/Sub.

"If you’re building a simple API (a small set of functions to be accessed via HTTP or Cloud Pub/Sub), we recommend using Cloud Functions."

Ref: https://cloud.google.com/serverless-options

https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints This list constraint defines the set of domains that email addresses added to Essential Contacts can have. By default, email addresses with any domain can be added to Essential Contacts. The allowed/denied list must specify one or more domains of the form @example.com. If this constraint is active and configured with allowed values, only email addresses with a suffix matching one of the entries from the list of allowed domains can be added in Essential Contacts. This constraint has no effect on updating or removing existing contacts. constraints/essentialcontacts.allowedContactDomains

gcloud compute ssh ensures that the user’s public SSH key is present in the project’s metadata. If the user does not have a public SSH key, one is generated using ssh-keygen and added to the project’s metadata. This is similar to the other option where we copy the key explicitly to the project’s metadata but here it is done automatically for us. There are also security benefits with this approach. When we use gcloud compute ssh to connect to Linux instances, we are adding a layer of security by storing your host keys as guest attributes. Storing SSH host keys as guest attributes improve the security of your connections by helping to protect against vulnerabilities such as man-in-the-middle (MITM) attacks. On the initial boot of a VM instance, if guest attributes are enabled, Compute Engine stores your generated host keys as guest attributes.

Compute Engine then uses these host keys that were stored during the initial boot to verify all subsequent connections to the VM instance.

Ref: https://cloud.google.com/compute/docs/instances/connecting-to-instance

Ref: https://cloud.google.com/sdk/gcloud/reference/compute/ssh

IAP controls access to your App Engine apps and Compute Engine VMs running on Google Cloud. It leverages user identity and the context of a request to determine if a user should be allowed access. IAP is a building block toward BeyondCorp, an enterprise security model that enables employees to work from untrusted networks without using a VPN.

By default, IAP uses Google identities and IAM. By leveraging Identity Platform instead, you can authenticate users with a wide range of external identity providers, such as:

Email/password

OAuth (Google, Facebook, Twitter, GitHub, Microsoft, etc.)

SAML

OIDC

Phone number

Custom

Anonymous

This is useful if your application is already using an external authentication system, and migrating your users to Google accounts is impractical.

https://cloud.google.com/iap/docs/using-tcp-forwarding#grant-permission

Comprehensive and Detailed In Depth Explanation:

The key requirement is to migrate applications that rely on hard-coded internal IP addresses without modifying the application code. To achieve this, the migrated VMs in Google Cloud need to retain their original internal IP addresses.

A. Non-overlapping CIDR ranges and new static IPs: This option requires changing the IP addresses of the migrated workloads, which would necessitate modifying the application code to reflect these new addresses. This violates a core requirement.

B. Migrating DNS and using ephemeral IPs: While migrating DNS can be beneficial in the long run, using ephemeral internal IP addresses for the migrated workloads means their IPs could change upon restart, breaking the hard-coded IP address dependencies.

C. Single subnet with Cloud NAT and static NAT IP: Cloud NAT allows instances without external IP addresses to access the internet, but it doesn't help in preserving the internal IP addresses that the applications use to communicate with each other. The internal IP addresses of the VMs would still be within the VPC subnet range and might conflict if they are the same as the on-premises IPs.

D. Same CIDR ranges and same static IPs: Creating a VPC with the same CIDR ranges as the on-premises network and assigning the same static internal IP addresses to the migrated workloads is the only way to ensure that the applications can continue to communicate using their hard-coded IP addresses without any code changes. This approach effectively extends the on-premises network's IP address space into Google Cloud (though without direct connectivity initially, as stated in the problem). Once the workloads are migrated, future steps can involve establishing connectivity (e.g., using VPN or Interconnect) if needed for hybrid scenarios.

Google Cloud Documentation References:

VPC Network Overview: https://cloud.google.com/vpc/docs/vpc - This document explains the fundamentals of VPC networks and their IP addressing. While it doesn't explicitly detail lift-and-shift scenarios with identical IP ranges without connectivity, it lays the groundwork for understanding VPC configuration.

Considerations for planning IP address ranges: https://cloud.google.com/vpc/docs/subnets#ip-ranges - This section discusses IP address planning, and while overlapping ranges are generally discouraged for connected networks, for isolated migration scenarios as described, it's a necessary step to avoid application changes. The problem statement explicitly says the environments are not connected during the initial migration.

===========

https://cloud.google.com/sdk/gcloud

https://cloud.google.com/sdk/docs/configurations#multiple_configurations

"Run gcloud configurations list to start the Compute Engine instances". How the heck are you expecting to "start" GCE instances doing "configuration list".

Each gcloud configuration has a 1 to 1 relationship with the region (if a region is defined). Since we have two different regions, we would need to create two separate configurations using gcloud config configurations create

Ref: https://cloud.google.com/sdk/gcloud/reference/config/configurations/create

Secondly, you can activate each configuration independently by running gcloud config configurations activate [NAME]

Ref: https://cloud.google.com/sdk/gcloud/reference/config/configurations/activate

Finally, while each configuration is active, you can run the gcloud compute instances start [NAME] command to start the instance in the configurations region.

https://cloud.google.com/sdk/gcloud/reference/compute/instances/start

C:\GCP\appeng>gcloud config list

[core]

account = xxx@gmail.com

disable_usage_reporting = False

project = my-first-demo-xxxx

https://cloud.google.com/endpoints/docs/openapi/troubleshoot-gce-deployment

You can create two configurations – one for the development project and another for the production project. And you do that by running “gcloud config configurations create” command.

https://cloud.google.com/sdk/gcloud/reference/config/configurations/create

In your custom script, you can load these configurations one at a time and execute gcloud compute instances list to list Google Compute Engine instances in the project that is active in the gcloud configuration.

Ref: https://cloud.google.com/sdk/gcloud/reference/compute/instances/list

Once you have this information, you can export it in a suitable format to a suitable target e.g. export as CSV or export to Cloud Storage/BigQuery/SQL, etc

Create an instance template for the instances so VMs have same specs. Set the "˜Automatic Restart' to on to VM automatically restarts upon crash. Set the "˜On-host maintenance' to Migrate VM instance. This will take care of VM during maintenance window. It will migrate VM instance making it highly available Add the instance template to an instance group so instances can be managed.

• onHostMaintenance: Determines the behavior when a maintenance event occurs that might cause your instance to reboot.

• [Default] MIGRATE, which causes Compute Engine to live migrate an instance when there is a maintenance event.

• TERMINATE, which stops an instance instead of migrating it.

• automaticRestart: Determines the behavior when an instance crashes or is stopped by the system.

• [Default] true, so Compute Engine restarts an instance if the instance crashes or is stopped.

• false, so Compute Engine does not restart an instance if the instance crashes or is stopped.

Enabling automatic restart ensures that compute engine instances are automatically restarted when they crash. And Enabling Migrate VM Instance enables live migrates i.e. compute instances are migrated during system maintenance and remain running during the migration.

Automatic Restart  If your instance is set to terminate when there is a maintenance event, or if your instance crashes because of an underlying hardware issue, you can set up Compute Engine to automatically restart the instance by setting the automaticRestart field to true. This settingdoes not apply if the instance is taken offline through a user action, such as calling sudo shutdown, or during a zone outage.

Ref: https://cloud.google.com/compute/docs/instances/setting-instance-scheduling-options#autorestart

Enabling the Migrate VM Instance option migrates your instance away from an infrastructure maintenance event, and your instance remains running during the migration.Your instance might experience a short period of decreased performance, although generally, most instances should not notice any difference. This is ideal for instances that require constant uptime and can tolerate a short period of decreased performance.

Ref: https://cloud.google.com/compute/docs/instances/setting-instance-scheduling-options#live_migrate

Creating or upgrading a cluster by specifying the version as latest does not provide automatic upgrades. Enable node auto-upgrades to ensure that the nodes in your cluster are up-to-date with the latest stable version.

https://cloud.google.com/kubernetes-engine/versioning-and-upgrades

Node auto-upgrades help you keep the nodes in your cluster up to date with the cluster master version when your master is updated on your behalf. When you create a new cluster or node pool with Google Cloud Console or the gcloud command, node auto-upgrade is enabled by default.

Ref: https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades

You can use the Google Cloud Pricing Calculator to total the estimated monthly costs for each GCP product. You dont incur any charges for doing so.

Ref: https://cloud.google.com/products/calculator

https://medium.com/google-cloud/how-to-deploy-cassandra-and-connect-on-google-cloud-platform-with-a-few-clicks-11ee3d7001d1

https://cloud.google.com/blog/products/databases/open-source-cassandra-now-managed-on-google-cloud

https://cloud.google.com/marketplace

You can deploy Cassandra as a Service, called Astra, on the Google Cloud Marketplace. Not only do you get a unified bill for all GCP services, your Developers can now create Cassandra clusters on Google Cloud in minutes and build applications with Cassandra as a database as a service without the operational overhead of managing Cassandra

App engine is a regional service, which means the infrastructure that runs your app(s) is located in a specific region and is managed by Google to be redundantly available across all the zoneswithin that region. Once an app engine deployment is created in a region, it cant be changed. The only way is to create a new project and create an App Engine instance in europe-west3, send all user traffic to this instance and delete the app engine instance in us-central.

Ref: https://cloud.google.com/appengine/docs/locations

https://cloud.google.com/sql/docs/mysql/backup-recovery/backups#can_i_export_a_backup

https://cloud.google.com/sql/docs/mysql/import-export#automating_export_operations

https://cloud.google.com/compute/docs/autoscaler#specifications

Autoscaling works independently from autohealing. If you configure autohealing for your group and an instance fails the health check, the autohealer attempts to recreate the instance. Recreating an instance can cause the number of instances in the group to fall below the autoscaling threshold (minNumReplicas) that you specify.

Since we need the application running at all times, we need a minimum 1 instance.

Only a single instance of the VM should run, we need a maximum 1 instance.

We want the application running at all times. If the VM crashes due to any underlying hardware failure, we want another instance to be added to MIG so that application can continue to serve requests. We can achieve this by enabling autoscaling. The only option that satisfies these three is Set autoscaling to On, set the minimum number of instances to 1, and then set the maximum number of instances to 1.

Ref: https://cloud.google.com/compute/docs/autoscaler

You write a lifecycle management rule in XML and push it to the bucket with gsutil lifecycle set config-xml-file. is not right.

gsutil lifecycle set enables you to set the lifecycle configuration on one or more buckets based on the configuration file provided. However, XML is not a valid supported type for the configuration file.

Ref: https://cloud.google.com/storage/docs/gsutil/commands/lifecycle

Write a script that runs gsutil ls -lr gs://myapp-gcp-ace-logs/ to find and remove items older than 90 days. Repeat this process every morning. is not right.

This manual approach is error-prone, time-consuming and expensive. GCP Cloud Storage provides lifecycle management rules that let you achieve this with minimal effort.

Write a script that runs gsutil ls -l gs://myapp-gcp-ace-logs/ to find and remove items older than 90 days. Schedule the script with cron. is not right.

This manual approach is error-prone, time-consuming and expensive. GCP Cloud Storage provides lifecycle management rules that let you achieve this with minimal effort.

Write a lifecycle management rule in JSON and push it to the bucket with gsutil lifecycle set config-json-file. is the right answer.

You can assign a lifecycle management configuration to a bucket. The configuration contains a set of rules which apply to current and future objects in the bucket. When an object meets the criteria of one of the rules, Cloud Storage automatically performs a specified action on the object. One of the supported actions is to Delete objects. You can set up a lifecycle management to delete objects older than 90 days. gsutil lifecycle set enables you to set the lifecycle configuration on the bucket based on the configuration file. JSON is the only supported type for the configuration file. The config-json-file specified on the command line should be a path to a local file containing the lifecycle configuration JSON document.

Ref: https://cloud.google.com/storage/docs/gsutil/commands/lifecycle

Ref: https://cloud.google.com/storage/docs/lifecycle

When you intially click on Monitoring(Stackdriver Monitoring) it creates a workspac(a stackdriver account) linked to the ACTIVE(CURRENT) Project from which it was clicked.

Now if you change the project and again click onto Monitoring it would create an another workspace(a stackdriver account) linked to the changed ACTIVE(CURRENT) Project, we don't want this as this would not consolidate our result into a single dashboard(workspace/stackdriver account).

If you have accidently created two diff workspaces merge them under Monitoring > Settings > Merge Workspaces > MERGE.

If we have only one workspace and two projects we can simply add other GCP Project under

Monitoring > Settings > GCP Projects > Add GCP Projects.

https://cloud.google.com/monitoring/settings/multiple-projects

Nothing about groupshttps://cloud.google.com/monitoring/settings?hl=en

Cloud Debugger helps inspect the state of an application, at any code location, without stopping or slowing down the running app //https://cloud.google.com/debugger/docs

When encountering a quotaExceeded error in BigQuery, you should follow these steps to diagnose and mitigate the issue:

Understand the Error:

The error message indicates that a quota was exceeded (either a short-term rate limit or a longer-term limit).

The response payload contains information about which quota was reached.

Quotas can fall into two categories:

rateLimitExceeded: Short-term limits. Retry the operation after a few seconds using exponential backoff.

quotaExceeded: Longer-term limits. Wait 10 minutes or longer before retrying the operation.

Search Errors in Cloud Audit Logs (Option A):

Cloud Audit Logs provide detailed information about API requests and responses.

By searching the logs, you can identify the specific API call that triggered the quotaExceeded error.

This helps you understand which resource or operation exceeded the quota.

View Errors in Cloud Monitoring (Option C):

Cloud Monitoring (formerly known as Stackdriver) provides insights into your Google Cloud resources.

Check the monitoring dashboard for any alerts related to BigQuery quotas.

You can set up custom monitoring rules to track specific quotas and receive notifications.

Other Options:

B. Configure Cloud Trace: Cloud Trace is used for performance analysis and latency tracking. It’s not directly related to quota issues.

D. Use Information Schema Views: Information schema views provide metadata about your datasets and tables but won’t help diagnose quota errors.

E. Use BigQuery Bl Engine: There is no such tool called “BigQuery Bl Engine.” This option is invalid.

Remember that some quotas replenish incrementally over a 24-hour period, so you don’t always need to wait a full 24 hours after reaching the limit. Ifyou consistently hit longer-term quotas, consider workload optimization or requesting a quota increase

https://cloud.google.com/billing/docs/how-to/export-data-bigquery "Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify."

The key to understand the requirement is : "The objects are written once and accessed frequently for 30 days"

Standard Storage

Standard Storage is best for data that is frequently accessed ("hot" data) and/or stored for only brief periods of time.

Archive Storage

Archive Storage is the lowest-cost, highly durable storage service for data archiving, online backup, and disaster recovery. Unlike the "coldest" storage services offered by other Cloud providers, your data is available within milliseconds, not hours or days. Archive Storage is the best choice for data that you plan to access less than once a year.

https://cloud.google.com/storage/docs/storage-classes#standard

cell phones are sending UDP packets and the only that can receive that type of traffic is a External Network TCP/UDPhttps://cloud.google.com/load-balancing/docs/network

https://cloud.google.com/load-balancing/docs/choosing-load-balancer#lb-decision-tree

Google Cloud Storage Triggers

Cloud Functions can respond to change notifications emerging from Google Cloud Storage. These notifications can be configured to trigger in response to various events inside a bucket—object creation, deletion, archiving and metadata updates.

Note: Cloud Functions can only be triggered by Cloud Storage buckets in the same Google Cloud Platform project.

Event types

Cloud Storage events used by Cloud Functions are based on Cloud Pub/Sub Notifications for Google Cloud Storage and can be configured in a similar way.

Supported trigger type values are:

google.storage.object.finalize

google.storage.object.delete

google.storage.object.archive

google.storage.object.metadataUpdate

Object Finalize

Trigger type value: google.storage.object.finalize

This event is sent when a new object is created (or an existing object is overwritten, and a new generation of that object is created) in the bucket.

https://cloud.google.com/functions/docs/calling/storage#event_types

Google Cloud provides Cloud Audit Logs, which is an integral part of Cloud Logging. It consists of two log streams for each project: Admin Activity and Data Access, which are generated by Google Cloud services to help you answer the question of who did what, where, and when? within your Google Cloud projects.

Ref: https://cloud.google.com/iam/docs/job-functions/auditing#scenario_external_auditors

https://cloud.google.com/iam/docs/understanding-roles#billing-roles

https://cloud.google.com/sdk/gcloud/reference/compute/networks/subnets/expand-ip-range

gcloud compute networks subnets expand-ip-range - expand the IP range of a Compute Engine subnetwork gcloud compute networks subnets expand-ip-range NAME --prefix-length=PREFIX_LENGTH [--region=REGION] [GCLOUD_WIDE_FLAG …]

We need to apply the GCP Best practices. roles/browser Browser Read access to browse the hierarchy for a project, including the folder, organization, and IAM policy. This role doesn't include permission to view resources in the project.https://cloud.google.com/iam/docs/understanding-roles

Keyword, Version, traffic splitting, App Engine supports traffic splitting for versions before releasing.

https://cloud.google.com/logging/docs/api/v2/resource-list

GKE Containers have more log than GKE Cluster Operations:

-GKE Containe:

cluster_name: An immutable name for the cluster the container is running in.

namespace_id: Immutable ID of the cluster namespace the container is running in.

instance_id: Immutable ID of the GCE instance the container is running in.

pod_id: Immutable ID of the pod the container is running in.

container_name: Immutable name of the container.

zone: The GCE zone in which the instance is running.

VS

-GKE Cluster Operations

project_id: The identifier of the GCP project associated with this resource, such as "my-project".

cluster_name: The name of the GKE Cluster.

location: The location in which the GKE Cluster is running.

Adding an API as a type provider

This page describes how to add an API to Google Cloud Deployment Manager as a type provider. To learn more about types and type providers, read the Types overview documentation.

A type provider exposes all of the resources of a third-party API to Deployment Manager as base types that you can use in your configurations. These types must be directly served by a RESTful API that supports Create, Read, Update, and Delete (CRUD).

If you want to use an API that is not automatically provided by Google with Deployment Manager, you must add the API as a type provider.

https://cloud.google.com/deployment-manager/docs/configuration/type-providers/creating-type-provider

The pending Pods resource requests are too large to fit on a single node of the cluster. Too many Pods are already running in the cluster, and there are not enough resources left to schedule the pending Pod. is the right answer.

When you have a deployment with some pods in running and other pods in the pending state, more often than not it is a problem with resources on the nodes. Heres a sample output of this use case. We see that the problem is with insufficient CPU on the Kubernetes nodes so we have to either enable auto-scaling or manually scale up the nodes.

https://cloud.google.com/kubernetes-engine/docs/concepts/deployment

Coldline Storage is the perfect service to store audit logs from all the projects and is very cost-efficient as well. Coldline Storage is a very low-cost, highly durable storage service for storing infrequently accessed data.

1. In Stackdriver Logging, create a filter to view only Compute Engine logs. 2. Click Create Export. 3. Choose BigQuery as Sink Service, and the platform-logs dataset as Sink Destination.

https://cloud.google.com/deployment-manager/docs/configuration/create-basic-configuration

What is Google Cloud Memorystore?

Overview. Cloud Memorystore for Redis is a fully managed Redis service for Google Cloud Platform. Applications running on Google Cloud Platform can achieve extreme performance by leveraging the highly scalable, highly available, and secure Redis service without the burden of managing complex Redis deployments.

https://cloud.google.com/run/docs/tutorials/gcloud

https://cloud.google.com/resource-manager/docs/creating-managing-projects

https://cloud.google.com/iam/docs/understanding-roles#primitive_roles

You can shut down projects using the Cloud Console. When you shut down a project, this immediately happens: All billing and traffic serving stops, You lose access to the project, The owners of the project will be notified and can stop the deletion within 30 days, The project will be scheduled to be deleted after 30 days. However, some resources may be deleted much earlier.

https://www.cloudskillsboost.google/focuses/1035?parent=catalog#:~:text=custom%20role%20at%20the%20organization%20level

The best option is to instruct the external consultant to generate an SSH key pair, and request the public key from the consultant. Then, add the public key to the instance yourself, and have the consultant access the instance through SSH with their private key. This way, you can grant theconsultant access to the instance without requiring a Google account or exposing the instance’s public IP address. This option also follows the best practice of using user-managed SSH keys instead of service account keys for SSH access1.

Option A is not feasible because the external consultant does not have a Google account, and therefore cannot use Identity-Aware Proxy (IAP) to access the instance. IAP requires the user to authenticate with a Google account and have the appropriate IAM permissions to access the instance2. Option B is not secure because it exposes the instance’s public IP address, which can increase the risk of unauthorized access or attacks. Option D is not correct because it reverses the roles of the public and private keys. The public key should be added to the instance, and the private key should be kept by the consultant. Sharing the private key with anyone else can compromise the security of the SSH connection3.

1: https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys

2: https://cloud.google.com/iap/docs/using-tcp-forwarding

3: https://cloud.google.com/compute/docs/instances/connecting-advanced#sshbetweeninstances

https://cloud.google.com/architecture/best-practices-vpc-design#shared-service Cloud VPN is another alternative. Because Cloud VPN establishes reachability through managed IPsec tunnels, it doesn't have the aggregate limits of VPC Network Peering. Cloud VPN uses a VPN Gateway for connectivity and doesn't consider the aggregate resource use of the IPsec peer. The drawbacks of Cloud VPN include increased costs (VPN tunnels and traffic egress), management overhead required to maintain tunnels, and the performance overhead of IPsec.

According to the Google Cloud documentation, you can use organization policy constraints to control the creation and expiration of service account keys. The constraints are:

constraints/iam.allowServiceAccountKeyCreation: This constraint allows you to specify which projects or folders can create service account keys. You can set the value to true or false, or use a condition to apply the constraint to specific service accounts. By setting this constraint to false for the organization and adding an exception for the pj-sa project, you can prevent developers from creating service account keys in other projects.

constraints/iam.serviceAccountKeyMaxLifetime: This constraint allows you to specify the maximum lifetime of service account keys. You can set the value to a duration in seconds, such as 86400 for one day. By setting this constraint to 86400 for the organization, you can ensure that all service account keys expire after one day.

These constraints are recommended by Google Cloud as best practices to minimize the risk of service account key misuse or compromise. They also help you reduce the cost of managing service account keys, as you do not need to implement a custom solution to rotate or delete them.

1: Associate Cloud Engineer Certification Exam Guide | Learn - Google Cloud

5: Create and delete service account keys - Google Cloud

Organization policy constraints for service accounts

Signed URLs are used to give time-limited resource access to anyone in possession of the URL, regardless of whether they have a Google account.https://cloud.google.com/storage/docs/access-control/signed-urls

auto-provisioning = Attaches and deletes node pools to cluster based on the requirements. Hence creating a GPU node pool, and auto-scaling would be betterhttps://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-provisioning

https://cloud.google.com/network-connectivity/docs/vpn/concepts/best-practices

Cloud Run takes any container images and pairs great with the container ecosystem: Cloud Build, Artifact Registry, Docker. ... No infrastructure to manage: once deployed, Cloud Run manages your services so you can sleep well. Fast autoscaling. Cloud Run automatically scales up or down from zero to N depending on traffic.

https://cloud.google.com/run

Managed instance groups offer autoscaling capabilities that let you automatically add or delete instances from a managed instance group based on increases or decreases in load (CPU Utilization in this case). Autoscaling helps your apps gracefully handle increases intraffic and reduce costs when the need for resources is lower. You define the autoscaling policy and the autoscaler performs automatic scaling based on the measured load (CPU Utilization in this case). Autoscaling works by adding more instances to your instance group when there is more load (upscaling), and deleting instances when the need for instances is lowered (downscaling). Ref:https://cloud.google.com/compute/docs/autoscaler

Cloud Billing export to BigQuery enables you to export detailed Google Cloud billing data (such as usage, cost estimates, and pricing data) automatically throughout the day to a BigQuery dataset that you specify. Then you can access your Cloud Billing data from BigQuery for detailed analysis, or use a tool like Looker Studio to visualize your data.

The reason is that when you do health check, you want the VM to be working. Do the first check after initial setup time of 3 mins = 180 s < 200 s is reasonable.

The reason why our autoscaling is adding more instances than needed is that it checks 30 seconds after launching the instance and at this point, the instance isnt up and isnt ready to serve traffic. So our autoscaling policy starts another instance  again checks this after 30 seconds and the cycle repeats until it gets to the maximum instances or the instances launched earlier are healthy and start processing traffic  which happens after 180 seconds (3 minutes). This can be easily rectified by adjusting the initial delay to be higher than the time it takes for the instance to become available for processing traffic.So setting this to 200 ensures that it waits until the instance is up (around 180-second mark) and then starts forwarding traffic to this instance. Even after a cool out period, if the CPU utilization is still high, the autoscaler can again scale up but this scale-up is genuine and is based on the actual load.

Initial Delay Seconds  This setting delays autohealing from potentially prematurely recreating the instance if the instance is in the process of starting up. The initial delay timer starts when the currentAction of the instance is VERIFYING.

Ref: https://cloud.google.com/compute/docs/instance-groups/autohealing-instances-in-migs

Navigate to Stackdriver Logging and select resource.labels.project_id=*. is not right.

Log entries are held in Stackdriver Logging for a limited time known as the retention period  which is 30 days (default configuration). After that, the entries are deleted. To keep log entries longer, you need to export them outside of Stackdriver Logging by configuring log sinks.

Ref: https://cloud.google.com/blog/products/gcp/best-practices-for-working-with-google-cloud-audit-logging

Configure a Cloud Scheduler job to read from Stackdriver and store the logs in BigQuery. Configure the table expiration to 60 days. is not right.

While this works, it makes no sense to use Cloud Scheduler job to read from Stackdriver and store the logs in BigQuery when Google provides a feature (export sinks) that does exactly the same thing and works out of the box.

Ref: https://cloud.google.com/logging/docs/export/configure_export_v2

Create a Stackdriver Logging Export with a Sink destination to Cloud Storage. Create a lifecycle rule to delete objects after 60 days. is not right.

You can export logs by creating one or more sinks that include a logs query and an export destination. Supported destinations for exported log entries are Cloud Storage, BigQuery, and Pub/Sub.

Ref: https://cloud.google.com/logging/docs/export/configure_export_v2

Sinks are limited to exporting log entries from the exact resource in which the sink was created: a Google Cloud project, organization, folder, or billing account. If it makes it easier to exporting from all projects of an organication, you can create an aggregated sink that can export log entries from all the projects, folders, and billing accounts of a Google Cloud organization.

Ref: https://cloud.google.com/logging/docs/export/aggregated_sinks

Either way, we now have the data in Cloud Storage, but querying logs information from Cloud Storage is harder than Querying information from BigQuery dataset. For this reason, we should prefer Big Query over Cloud Storage.

Create a Stackdriver Logging Export with a Sink destination to a BigQuery dataset. Configure the table expiration to 60 days. is the right answer.

You can export logs by creating one or more sinks that include a logs query and an export destination. Supported destinations for exported log entries are Cloud Storage, BigQuery, and Pub/Sub.

Ref: https://cloud.google.com/logging/docs/export/configure_export_v2

Sinks are limited to exporting log entries from the exact resource in which the sink was created: a Google Cloud project, organization, folder, or billing account. If it makes it easier to exporting from all projects of an organication, you can create an aggregated sink that can export log entries from all the projects, folders, and billing accounts of a Google Cloud organization.

Ref: https://cloud.google.com/logging/docs/export/aggregated_sinks

Either way, we now have the data in a BigQuery Dataset. Querying information from a Big Query dataset is easier and quicker than analyzing contents in Cloud Storage bucket. As our requirement is to Quickly analyze the log contents, we should prefer Big Query over Cloud Storage.

Also, You can control storage costs and optimize storage usage by setting the default table expiration for newly created tables in a dataset. If you set the property when the dataset is created, any table created in the dataset is deleted after the expiration period. If you set the property after the dataset is created, only new tables are deleted after the expiration period.

For example, if you set the default table expiration to 7 days, older data is automatically deleted after 1 week.

Ref: https://cloud.google.com/bigquery/docs/best-practices-storage

Comprehensive and Detailed In Depth Explanation:

Let's evaluate each database option against the requirements: automatic scaling, multi-region support, and low latency for a growing social media platform:

A. Bigtable: Bigtable is a highly scalable NoSQL database designed for large analytical and operational workloads with low latency. It offers excellent horizontal scalability and can be deployed across multiple regions for high availability and lower latency for a global user base. However, it's a NoSQL database and might require significant changes to your existing PostgreSQL data model and application code.

B. BigQuery: BigQuery is a fully managed, serverless data warehouse optimized for analytical queries on large datasets. It's not designed for low-latency transactional workloads that a social media platformwould require for real-time user interactions. While it's globally available, its primary use case is not operational database needs.

C. Spanner: Spanner is a globally distributed, horizontally scalable relational database service with strong consistency. It offers automatic scaling, built-in multi-region and multi-continental configurations for high availability and low latency across a global user base, and supports standard SQL (with some extensions). This makes it a strong candidate for a rapidly growing platform needing scalability, global presence, and low latency. While it's not directly PostgreSQL, it offers a relational model and tools to aid migration.

D. Cloud SQL for PostgreSQL: Cloud SQL offers managed PostgreSQL instances with automatic scaling capabilities. It supports high availability within a region and cross-region read replicas for disaster recovery and read scaling. However, its multi-region capabilities for write operations and automatic scaling across regions are more limited compared to Spanner. For a rapidly growing platform with a primarily North American user base but future global expansion in mind and a need for low latency, Spanner's architecture is better suited for true multi-region write capabilities and consistent low latency globally.

Considering the requirements for automatic scaling, multi-region support for both reads and writes with low latency for a growing user base, Spanner is the most appropriate choice.

Google Cloud Documentation References:

Cloud Spanner Overview: https://cloud.google.com/spanner/docs/overview - This document highlights Spanner's global scalability, strong consistency, and multi-region capabilities.

Cloud Bigtable Overview: https://cloud.google.com/bigtable/docs/overview - While scalable and low-latency, it's a NoSQL database, which might require significant application changes.

BigQuery Overview: https://cloud.google.com/bigquery/docs/introduction - Focuses on analytics, not low-latency transactional workloads.

Cloud SQL for PostgreSQL Overview: https://cloud.google.com/sql/docs/postgres/overview - While it offers scaling and regional HA, its multi-region write capabilities are not as robust as Spanner's.

===========

Keywords, Financial data (large data) used globally, data stored and queried using relational structure (SQL), clients should get exact identical copies(Strong Consistency), Multiple region, low latency to end user, select storage option to minimize latency.

This answer is the most effective way to validate whether the service account used by the CI/CD server has the appropriate roles in the specific project. By checking the IAM roles assigned to the service account, you can see which permissions the service account has and which resources it can access. You can also check if the service account inherits any roles from the folder or organization levels, which may affect its access to the project. You can use the Google Cloud console, the gcloud command-line tool, or the IAM API to view the IAM roles of a service account.

The resource that you need to get access is in the other project.

roles/bigquery.dataViewer BigQuery Data Viewer

When applied to a table or view, this role provides permissions to:

Read data and metadata from the table or view.

This role cannot be applied to individual models or routines.

When applied to a dataset, this role provides permissions to:

Read the dataset's metadata and list tables in the dataset.

Read data and metadata from the dataset's tables.

When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.

GCP App Engine natively offers traffic splitting functionality between versions. You can use traffic splitting to specify a percentage distribution of traffic across two or more of the versions within a service. Splitting traffic allows you to conduct A/B testing between your versions and provides control over the pace when rolling out features.

Ref: https://cloud.google.com/appengine/docs/standard/python/splitting-traffic

This command correctly specifies the duration that the signed url should be valid for by using the -d flag. The default is 1 hour so omitting the -d flag would have also resulted in the same outcome. Times may be specified with no suffix (default hours), or with s = seconds, m = minutes, h = hours, d = days. The max duration allowed is 7d.

Ref: https://cloud.google.com/storage/docs/gsutil/commands/signurl

https://cloud.google.com/bigquery/docs/access-control#custom_roles

Custom roles enable you to enforce the principle of least privilege, ensuring that the user and service accounts in your organization have only the permissions essential to performing their intended functions.

Predefined roles

The following table describes Identity and Access Management (IAM) roles that are associated with Cloud Storage and lists the permissions that are contained in each role. Unless otherwise noted, these roles can be applied either to entire projects or specific buckets.

Storage Object Creator (roles/storage.objectCreator) Allows users to create objects. Does not give permission to view, delete, or overwrite objects.

https://cloud.google.com/storage/docs/access-control/iam-roles#standard-roles

https://cloud.google.com/compute/docs/instances/managing-instance-access

Choosing a region and zone You choose which region or zone hosts your resources, which controls where your data is stored and used. Choosing a region and zone is important for several reasons:

Handling failures

Distribute your resources across multiple zones and regions to tolerate outages. Google designs zones to be independent from each other: a zone usually has power, cooling, networking, and control planes that are isolated from other zones, and most single failure events will affect only a single zone. Thus, if a zone becomes unavailable, you can transfer traffic to another zone in the same region to keep your services running. Similarly, if a region experiences any disturbances, you should have backup services running in a different region. For more information about distributing your resources and designing a robust system, see Designing Robust Systems. Decreased network latency To decrease network latency, you might want to choose a region or zone that is close to your point of service.https://cloud.google.com/compute/docs/regions-zones#choosing_a_region_and_zone

From this source:https://cloud.google.com/billing/docs/how-to/custom-roles#permission_association_and_inheritance

"For example, associating a project with a billing account requires the billing.resourceAssociations.create permission on the billing account and also the resourcemanager.projects.createBillingAssignment permission on the project. This is because project permissions are required for actions where project owners control access, while billing account permissions are required for actions where billing account administrators control access. When both should be involved, both permissions are necessary."

https://cloud.google.com/compute/docs/instances/connecting-to-windows#remote-desktop-connection-app

https://cloud.google.com/compute/docs/instances/windows/generating-credentials

https://cloud.google.com/compute/docs/instances/connecting-to-windows#before-you-begin

gcloud deployment-manager deployments create creates deployments based on the configuration file. (Infrastructure as code). All the configuration related to the artifacts is in the configuration file. This command correctly creates a cluster based on the provided cluster.yaml configuration file.

Ref: https://cloud.google.com/sdk/gcloud/reference/deployment-manager/deployments/create

Understanding IAM custom roles

Key Point: Custom roles enable you to enforce the principle of least privilege, ensuring that the user and service accounts in your organization have only the permissions essential to performing their intended functions.

Basic concepts

Custom roles are user-defined, and allow you to bundle one or more supported permissions to meet your specific needs. Custom roles are not maintained by Google; when new permissions, features, or services are added to Google Cloud, your custom roles will not be updated automatically.

When you create a custom role, you must choose an organization or project to create it in. You can then grant the custom role on the organization or project, as well as any resources within that organization or project.

https://cloud.google.com/iam/docs/understanding-custom-roles#basic_concepts

Logged information Within Cloud Audit Logs, there are two types of logs: Admin Activity logs: Entries for operations that modify the configuration or metadata of a project, bucket,or object. Data Access logs: Entries for operations that modify objects or read a project, bucket, or object. There are several sub-types of data access logs: ADMIN_READ: Entries for operations that read the configuration or metadata of a project, bucket, or object. DATA_READ: Entries for operations that read an object. DATA_WRITE: Entries for operations that create or modify an object.https://cloud.google.com/storage/docs/audit-logs#types