PolarDB scheduled backup to OSS fails with insufficient permissions error, backup not created

We’ve configured scheduled backups from our PolarDB MySQL cluster to an OSS bucket for long-term retention and disaster recovery purposes. The backups were working fine for about two weeks, but suddenly started failing three days ago with an “insufficient permissions” error.

Our setup:

  • PolarDB MySQL 8.0 cluster in China East 2
  • Target OSS bucket in the same region
  • Scheduled daily backups at 02:00 AM
  • RAM role attached to PolarDB service

Error from PolarDB Console backup logs:


BackupTaskFailed: Access denied when writing to OSS
ErrorCode: AccessDenied
Bucket: prod-db-backups
RAM Role: AliyunPolarDBBackupRole

We haven’t changed any bucket policies or RAM permissions recently. The RAM role still appears to have the AliyunOSSFullAccess policy attached. Manual backups through the console also fail with the same error. Has anyone encountered this issue? We need to restore the backup capability urgently as our compliance requires daily backups with 90-day retention.

The issue is clear - your bucket policy is blocking the PolarDB service. Let me walk you through the proper configuration.

First, your RAM role trust policy looks correct. The role trusts the PolarDB service principal, which is good. However, you should create a custom policy instead of using OSSFullAccess for better security. Here’s a minimal permission policy for PolarDB backups:

{
  "Version": "1",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["oss:PutObject", "oss:GetObject", "oss:DeleteObject"],
    "Resource": ["acs:oss:*:*:prod-db-backups/polardb-backups/*"]
  }]
}

Now for the bucket policy fix - you need to modify your existing bucket policy to allow access from the RAM role. Replace your current bucket policy with this:

{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"RAM": ["acs:ram::YOUR_ACCOUNT_ID:role/AliyunPolarDBBackupRole"]},
      "Action": ["oss:PutObject", "oss:GetObject"],
      "Resource": ["acs:oss:*:*:prod-db-backups/polardb-backups/*"]
    },
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": "oss:*",
      "Resource": ["acs:oss:*:*:prod-db-backups/*"],
      "Condition": {
        "IpAddress": {"acs:SourceIp": ["!YOUR_OFFICE_IP_RANGE"]},
        "StringNotEquals": {"acs:PrincipalARN": ["acs:ram::YOUR_ACCOUNT_ID:role/AliyunPolarDBBackupRole"]}
      }
    }
  ]
}

Key points about this configuration:

  1. RAM Role Permissions: The first statement explicitly allows the PolarDB backup RAM role to write and read objects in the polardb-backups prefix. This ensures PolarDB service can perform backup operations.

  2. Bucket Policy Security: The second statement maintains your security team’s IP restriction for regular users while creating an exception for the PolarDB RAM role. The condition uses StringNotEquals to exclude the role from the IP restriction.

  3. Principle of Least Privilege: Both policies grant only the necessary permissions (PutObject, GetObject) rather than full access. For backups, you don’t need ListBucket or other administrative permissions.

  4. Resource Scoping: Notice we’re using a specific prefix (polardb-backups/*) to limit where PolarDB can write. This prevents accidental overwrites of other data in the bucket.

Implementation steps:

  1. Go to RAM Console > Roles > AliyunPolarDBBackupRole
  2. Remove AliyunOSSFullAccess and add the custom policy above
  3. Go to OSS Console > prod-db-backups > Access Control > Bucket Policy
  4. Replace the current policy with the one above (update YOUR_ACCOUNT_ID and YOUR_OFFICE_IP_RANGE)
  5. In PolarDB Console, verify the RAM role is still attached to your cluster
  6. Trigger a manual backup to test

For database recovery scenarios, this configuration ensures you can restore from OSS backups while maintaining security controls. The RAM role approach is much more secure than using IP whitelisting alone because it provides identity-based access control that works regardless of where the PolarDB service IPs originate.

After applying these changes, your scheduled backups should resume successfully. Monitor the first few backup jobs to confirm everything works as expected.

I had a similar issue last month. Check if your OSS bucket has any bucket policies that might be blocking the access. Even if the RAM role has OSSFullAccess, a restrictive bucket policy can override it. Go to OSS Console > your bucket > Access Control > Bucket Policy and see if there are any deny rules that might have been added.

Here’s our current RAM role trust policy - it looks pretty basic:

{
  "Statement": [{
    "Action": "sts:AssumeRole",
    "Effect": "Allow",
    "Principal": {"Service": ["polardb.aliyuncs.com"]}
  }]
}

The role has AliyunOSSFullAccess system policy attached. What changes do we need to make to the bucket policy to allow PolarDB service while maintaining security?

Yes, that was our security hardening initiative. We’re tightening bucket access across all projects. For PolarDB backup scenarios, you need to whitelist the service endpoint IPs or better yet, use RAM role-based access with proper conditions instead of IP restrictions. The bucket policy should trust the RAM role’s AssumeRole action from PolarDB service principal. Can you share your current RAM role trust policy? We can help adjust both the role and bucket policy to work together securely.

Perfect! This worked exactly as described. I implemented the custom RAM policy and updated the bucket policy following your instructions. The manual backup test succeeded immediately, and last night’s scheduled backup completed without errors.

A few observations:

  • The custom policy is much cleaner than using the system OSSFullAccess policy
  • The bucket policy now clearly shows which role has access and why
  • Our security team is happy because the IP restrictions remain in place for regular users
  • PolarDB backup logs now show successful writes to OSS with proper authentication

For anyone else facing this issue, make sure to:

  1. Check both RAM role permissions AND bucket policies - they both matter
  2. Use the specific prefix path in your policies to scope permissions properly
  3. Test with a manual backup before relying on scheduled backups
  4. Keep the backup retention policy aligned between PolarDB and OSS lifecycle rules

The key insight here is understanding that bucket policies can override RAM permissions, so you need both configured correctly. The condition-based exception in the bucket policy is particularly elegant - it maintains security while allowing service-to-service access. Thanks everyone for the help, especially the detailed solution with the proper policy configurations!

Thanks for the suggestion. I checked the bucket policy and found something interesting - there’s a policy that was added on March 12th (the day before backups started failing) that restricts write access to specific IP addresses. It looks like someone from the security team added it without coordinating with us. The policy allows writes only from our office IP range, but PolarDB service uses Alibaba Cloud internal IPs which aren’t in that range. This explains why it worked before and suddenly stopped.