How to build an AWS and Azure cloud inventory with Python

Getting a list of cloud resources programmatically sounds like a straightforward task. Authenticate with the provider, call an inventory endpoint, and process the response. In practice, neither AWS nor Azure offers one API that returns a complete, useful picture of an environment.

Building an AWS cloud inventory means calling dozens of service APIs. Most resources must be discovered separately in every enabled region, while global services such as IAM and Route 53 need different handling. Azure has its own SDKs, authentication flow, subscription model, resource identifiers, and response structures. Even after collecting the data, the two providers describe similar infrastructure in very different ways.

We ran into this problem while building Cloustral. Before we could visualize cloud architecture, evaluate governance rules, or flag resources that needed attention, we first needed a dependable way to collect AWS and Azure resources and turn them into one consistent inventory.

Why cloud resource collection gets complicated

A cloud inventory collector has to solve much more than authentication. Every service makes slightly different assumptions about pagination, regions, identifiers, tags, status values, relationships, and permissions. Some APIs return everything in one response. Others require paginators, follow-up calls, or separate requests for tags and configuration details.

The differences become more visible when an application supports multiple providers. An AWS VPC, an Azure virtual network, an EC2 instance, and an Azure virtual machine do not share a native schema, even though an inventory application needs to search, display, and connect them in comparable ways.

A production collector also needs to expect incomplete access. One missing permission, an unavailable regional service, or a disabled API should not erase hundreds of resources that were collected successfully elsewhere.

That leaves application teams maintaining the same infrastructure plumbing:

• Discovering and iterating over enabled AWS regions.
• Calling regional and global services with the correct scope.
• Handling pagination and service-specific failure modes.
• Running independent API calls concurrently without overwhelming provider APIs.
• Normalizing AWS and Azure responses into a common resource model.
• Preserving useful partial results when an individual collector fails.

Introducing Cloud Harvester

We extracted that collection layer from the problem we were solving and created Cloud Harvester: a Python package for collecting AWS and Azure cloud inventory through one entry point. It is publicly available on PyPI and can be installed with pip, Poetry, or uv.

pip install cloud-harvester
poetry add cloud-harvester
uv add cloud-harvester

Cloud Harvester accepts a boto3 session for AWS and Azure credentials plus a subscription ID for Azure. Its collect() function fans out to dedicated collectors across compute, containers and serverless, networking and edge, storage, databases, identity and security, and observability services.

The same call can collect AWS and Azure resources at once. This example injects an AWS session and Azure service principal credentials directly:

import boto3
from azure.identity import ClientSecretCredential
from cloud_harvester import collect

# AWS: static credentials (replace with real values)
aws_session = boto3.Session(
    aws_access_key_id="FAKEAWSACCESSKEY123",
    aws_secret_access_key="FAKEAWSSECRETKEY456",
)

# Azure: service principal credentials (replace with real values)
azure_credential = ClientSecretCredential(
    tenant_id="00000000-0000-0000-0000-000000000000",
    client_id="11111111-1111-1111-1111-111111111111",
    client_secret="fake-azure-client-secret",
)
azure_subscription_id = "22222222-2222-2222-2222-222222222222"

# Collect from both providers with injected sessions/credentials
result = collect(
    providers=["aws", "azure"],
    aws_session=aws_session,
    azure_credential=azure_credential,
    azure_subscription_id=azure_subscription_id,
)

for res in result.resources:
    print(res.to_dict())

for err in result.errors:
    print(err.provider, err.collector, err.region, err.error_code, err.error_message)

If no AWS region list is supplied, the package discovers and scans every enabled region. You can provide an explicit list when you only need selected regions, and tune max_workers when you want tighter control over concurrency and API pressure.

One resource model for AWS and Azure

Collecting responses is only half of the job. Applications should not need separate inventory logic for every provider and service, so Cloud Harvester converts every discovered item into the same Resource model.

The normalized fields cover identity, provider, resource kind, service, name, region, status, tags, network and subnetwork placement, addresses, and relationships. This gives downstream code a stable structure for searching resources, building topology graphs, evaluating policies, or exporting inventory data.

Normalization does not mean throwing useful information away. Each Resource also retains the original provider payload, so applications can inspect service-specific details whenever the common model is not enough.

Partial inventory is better than no inventory

Cloud permissions are rarely uniform across a real environment. A collector may be denied access to one AWS service in one region while every other request succeeds. Treating that single error as a failure of the entire collection would make the result far less useful.

Cloud Harvester therefore returns a CollectionResult containing both resources and errors. Successfully collected resources remain available, while each failed collector produces a structured error with its provider, collector name, region, error code, and message. An application can use the inventory immediately and show exactly which part could not be accessed.

Built for a real multi-cloud inventory

Cloud Harvester grew out of the inventory work behind Cloustral, where cloud resources need to support architecture views, relationships, governance rules, findings, and ongoing synchronization. Extracting the collection layer gave us a smaller boundary to test and improve, while making the same foundation available to other Python projects.

If you are building a cloud management platform, an internal asset inventory, a governance tool, or a reporting workflow, you can start with normalized AWS and Azure resources instead of first writing and maintaining dozens of SDK integrations.

Start collecting cloud resources with Python

Cloud Harvester requires Python 3.12 or newer. Configure read access for the cloud environments you want to inspect, install the package, and call collect() with the providers and credentials your application uses.

The package, installation instructions, credential requirements, and complete quickstart are available on the Cloud Harvester PyPI page.