Validate Boto3 Models with Pydantic
This is the second blog post about the AWS resource validator package. In the previous blog we learned about various resource strings (like names, ARNs) validation and generation using Python classes that were automatically generated from the botocore repository.
The second time around we’re going to take a closer look at boto3 and one-up our validator game. Instead of relying on callable functions like .validate() and .generate(), we’re going to use Pydantic and it’s auto-validation powers to create boto3 payloads and, even better, validate and parse into Python objects it’s responses.
The Boto3 REST Client
If you ever had to work with boto3, you are familiar with figuring out and constructing compatible payloads using primitive strings and maps. Even worse, you to work with multi-layered maps when parsing and extracting information from the responses of the various boto3 clients. Yes, you can wrap them with your own classes and parse them into objects, or even work with the auto-generated TypeDef models from AWS’ own stub libraries, but those solve only half the problem.
First, if you want to go the route of creating your own classes, you still need to first figure out the response layout. You either capture one using debug (or by printing), and then add fields that are of interest to you to your class. You can also use the boto stubs to figure out what a specific response might look like — and then manually write out your class.
Furthermore — you have to repeat this process, again and again, for each response and for each boto3 client that you utilize. The “create dynamoDB table” response from the dynamodb client will not necessarily be the same as the response of “create Hosted Zone” from the route53 client.
You also need to keep track and update those classes in case there are changes or updates to the various boto3 clients that you need.
We’ve only discussed responses from boto3 clients until now — but it’s the same for any payloads they might require.
Pydantic Boto3 Classes
Now let's consider a simpler, more efficient version of the flow that we explored in the previous paragraph. Instead of creating a Python class, or manually crafting a Dict — you can just select the matching Pydantic class, and populate it’s fields. You don’t even have to guess which ones are mandatory — Pydantic provides this information to you and validates your input on-the-fly.
The classes are conveniently grouped into files representing the service name, so that you can easily find your required class.
Once you’ve crafted the payload and triggered your API — parsing the response is a piece of cake. Using Pydantic’s build-in parser and validator, you can just thrown the JSON that you got back from the boto3 client right at it — and let it instantiate, validate and populate the class fields for you.
You can even pair this practice with the extended validation offered by the aws_resource_validator package to make sure that those strings that you got in your response actually represent valid ARNs, resource named and IDs.
How It’s Done
The aws_resource_validator project is open source and you can take a look at our code if you are interested in the technological solution. I’ll briefly cover some technical aspects of this code generator.
We’ve used the available boto3-stubs that are themselves, in term, are generated Python classes. It was easy to get them into the project — simply declare them as a required dependency — and they are in your virtual environment.
It was then just a matter of parsing the type definitions and the literals that are used in each type definition. This was achieved by a not-so-simple regex, which is basically the whole “logic” behind this Pydantic generator.
Once the type definition is correctly captured by the regex filter, we can convert it into a Pydantic-compatible class, and just write it, as a string, into a .py file.
Viola! You have an importable Python library with Pydantic classes.
We’ve also included the Literals from the boto3-stubs, so that you won’t have to guess which values are acceptable in each field. They also allow us to leverage the validation powers of Pydantic to make sure that the response objects hold the correct values in each field.
A Practical Example
If we’d like to see how this can actually simplify and accelerate your development, we can consider the following code examples. The first one would simulate working with the aws_resource_validator package, and the second one — without.
import boto3
dynamodb = boto3.client('dynamodb')
def list_dynamo_tables():
try:
response = dynamodb.list_tables()
table_names = response.get('TableNames', [])
return table_names
except Exception as e:
print(f"Error listing tables: {e}")
return []
if __name__ == "__main__":
tables = list_dynamo_tables()
print("DynamoDB Tables:", tables)And now using the Pydantic classes:
import boto3
from aws_resource_validator.pydantic_models.dynamodb_classes import ListTablesOutputTypeDef
dynamodb = boto3.client('dynamodb')
def list_dynamo_tables():
return ListTablesOutputTypeDef(**dynamodb.list_tables()).TableNames
if __name__ == "__main__":
tables = list_dynamo_tables()
print("DynamoDB Tables:", tables)Now, lets consider a more complicated use case — creating a table.
from typing import List
import boto3
dynamodb = boto3.client('dynamodb')
def create_dynamo_table(table_name: str, attribute_definitions: List, key_schema: List):
try:
response = dynamodb.create_table(
TableName=table_name,
AttributeDefinitions=attribute_definitions,
KeySchema=key_schema
)
return response
except Exception as e:
print(f"Error creating table: {e}")
return None
if __name__ == "__main__":
table_name = "ExampleTable"
attribute_definitions = [
{
'AttributeName': 'Id',
'AttributeType': 'N'
}
]
key_schema = [
{
'AttributeName': 'Id',
'KeyType': 'HASH'
}
]
response = create_dynamo_table(table_name, attribute_definitions, key_schema)
if response:
print("Table created successfully:", response)And now with a generated Pydantic schema:
from aws_resource_validator.pydantic_models.dynamodb_classes import CreateTableInputRequestTypeDef, AttributeDefinitionTypeDef, KeySchemaElementTypeDef
from aws_resource_validator.pydantic_models.dynamodb_constants import ScalarAttributeTypeType, KeyTypeType
import boto3
dynamodb = boto3.client('dynamodb')
def create_dynamo_table(table_spec: CreateTableInputRequestTypeDef):
try:
response = dynamodb.create_table(**table_spec.model_dump())
return response
except Exception as e:
print(f"Error creating table: {e}")
return None
if __name__ == "__main__":
table_spec = CreateTableInputRequestTypeDef(
TableName="ExampleTable2",
AttributeDefinitions=[
AttributeDefinitionTypeDef(
AttributeName="Id",
AttributeType=ScalarAttributeTypeType.S
)
],
KeySchema=[
KeySchemaElementTypeDef(
AttributeName="Id",
KeyType=KeyTypeType.HASH
)
]
)
response = create_dynamo_table(table_spec)
if response:
print("Table created successfully:", response)See how we took out all the guesswork and working with strings? You can work with well defined classes, and be confident that your payload is valid.
We can clearly see that the generated classes found in aws_resource_validator can make you code cleaner, leaner and more maintainable. You don’t have to “guess” the keys of the response, check the documentation — or even validate. The Pydantic model does everything for you.
Recap
If you are working with boto3 clients in your AWS Python code — you’re pretty much guaranteed to benefit from the generated Pydantic classes in aws_resource_validator. As demonstrated, they are incredibly useful and can save you countless hours of writing, maintaining and validating your own custom schemas.
As always, thanks and love to my beautiful wife and talented DevOps Architect Yafit Tupman, my development partner & my personal DevOps.
If you have found an issue and want it fixed — feel free to open an issue on GitHub.
We also accept code contributions! See our code contribution guide for information about submitting your first PR.
