We’re experiencing API access denied errors when running Dataflow ETL jobs that need to access external APIs from within our VPC Service Controls perimeter. The jobs were working fine until we enabled the service perimeter for enhanced security.
The error we’re seeing:
Error 403: Request violates VPC Service Controls
Access to API denied by service perimeter
at DataflowRunner.execute(line 234)
Our service perimeter is configured to protect BigQuery and Cloud Storage, but now Dataflow can’t reach the necessary APIs for our data ingestion pipeline. We’ve checked IAM roles for the Dataflow service account and everything looks correct there. I’m not sure if this is an Access Context Manager configuration issue or if we need to adjust the perimeter rules specifically for Dataflow API access. Any guidance on the proper configuration would be greatly appreciated.
No, don’t add policyEditor to the service account - that’s for administrators managing perimeters. What you need is to configure egress rules in your service perimeter to allow outbound API calls. The service account itself needs roles/dataflow.worker and appropriate API access roles, but the perimeter configuration controls what traffic can flow through. You’ll want to create an egress policy that specifies the external APIs as allowed destinations.
This is a common issue when implementing VPC Service Controls. The perimeter is blocking API calls that Dataflow needs to make. You’ll need to configure ingress/egress rules in Access Context Manager to allow specific API access while maintaining security. Check which APIs your Dataflow jobs are calling and add them to the allowed services list in your perimeter configuration.
Have you verified the IAM roles include the Service Perimeter Admin role? The Dataflow service account needs proper permissions not just for the APIs themselves, but also to operate within the perimeter boundaries. Also check if you have any organization policies that might be restricting cross-project or cross-perimeter access.
Thanks for the responses. I checked and the service account has roles/dataflow.worker and roles/bigquery.dataEditor. Should I be adding roles/accesscontextmanager.policyEditor as well? I’m hesitant to grant too many permissions. Our Dataflow jobs need to call external REST APIs for enrichment during the ETL process.
I dealt with this exact scenario last month. The key is understanding that VPC Service Controls operates at the API level, not the network level. Your Dataflow workers are inside the perimeter trying to call APIs that are either outside or restricted. You need to explicitly allow those API calls in the perimeter config through Access Context Manager, separate from IAM permissions which just control who can do what.