There are three best practice areas for operational excellence in the cloud:

Prepare

Operate

Evolve

Operations teams need to understand their business and customer needs so they can effectively and efficiently support business outcomes. Operations create and use procedures to respond to operational events and validate their effectiveness to support business needs. Operations collect metrics that are used to measure the achievement of desired business outcomes.

Use Remote State: Remote state allows you to share the state of your infrastructure between all members of your team. This is necessary for collaboration and is more secure because only the changes are pulled and pushed, which means that sensitive parts of your state never have to be on a disk.

Enable State Locking: State locking helps to prevent any concurrent runs of Terraform that could lead to corruption of the state file or conflicts in the infrastructure changes. Many remote backends like AWS S3 (when used with DynamoDB), Azure Blob Storage, Google Cloud Storage, etc., support state locking.

Secure Your Backend: The state file can contain sensitive information, so it’s essential to secure it. Use encryption at rest if it’s supported by the backend. Also, control access to the backend using appropriate IAM roles and policies.

Keep Different Environments Separate: You should have different state files for different environments like production, staging, development, etc. This can be achieved using workspaces or separate backend configurations.

Use Versioning: If your backend supports versioning (like AWS S3), enable it. It allows you to roll back to a previous version of the state file if something goes wrong.

Backup Your State File: Even though your state is stored remotely and possibly versioned, it’s still a good idea to occasionally backup your state file, especially before making significant changes.

Limit Access: The state file can include sensitive data, depending on your infrastructure. You should limit access to the state file to only those who absolutely need it.