This is an adaptation from an article published on the AWS Builder blog. You can read the full text here
In today's data ecosystem, agility and reliability are paramount. Integrating Dataddo with technologies like Apache Iceberg within the AWS environment allows organizations to establish a robust foundation for their analytics and AI operations.
1: Provisioning via AWS Marketplace
As an AWS Partner, Dataddo can be acquired directly through the AWS Marketplace. This significantly simplifies billing and vendor management, allowing for a rapid setup that includes the automatic creation of IAM roles to ensure secure access to Amazon S3 and the AWS Glue Data Catalog.
2: Connecting Data Sources
The first operational step is connecting your source systems. Dataddo offers a wide range of connectors—both pre-configured and custom—for SaaS applications, ERPs, databases, and APIs. The platform handles authentication management and ongoing connection maintenance as source systems evolve.
3: Configuring the Iceberg Destination in Amazon S3
Once sources are connected, they are configured to write directly into Apache Iceberg tables within your own Amazon S3 environment. Dataddo allows you to align dataset structures and update frequencies according to specific business needs.
Step 4: Registration in AWS Glue Data Catalog
To make the data discoverable and usable by other AWS analytics services, Iceberg tables must be registered in the AWS Glue Data Catalog. It is crucial to verify this registration in Amazon Athena to confirm that metadata, schemas, and partitions are visible and up to date.
Step 5: Transformation and Modeling with AWS Glue
With the data already in Iceberg format, AWS Glue jobs can be employed to enrich, aggregate, and model the information. This process applies the necessary business logic to generate curated datasets ready for final consumption.
Step 6: Validation and Monitoring
Data flow integrity is ensured through:
- AWS Glue Data Quality: To validate the accuracy and completeness of the information.
- Amazon CloudWatch: To set up alarms for process failures or performance degradation.
- Dataddo Dashboards: To monitor ingestion metrics and quality indicators directly from the platform.
Step 7: Data Consumption
Finally, the data is optimized and available to be queried and leveraged by the entire suite of AWS analytical services, closing the loop from raw source to business value.
Conclusion
By combining Dataddo’s seamless ingestion capabilities with the high-performance table format of Apache Iceberg on AWS, organizations can build a future-proof data lake that is both open and scalable. This integration not only streamlines the journey from disparate SaaS and database sources to a centralized S3 environment but also ensures that data remains reliable, well-governed through the AWS Glue Data Catalog, and ready for advanced analytics. Ultimately, this architecture empowers teams to spend less time on pipeline maintenance and more time extracting actionable insights from their data.
