A serverless approach to relational data replication!

by · Published · Updated

       

A serverless approach to relational data replication!

I felt it challenging keeping cost & complexity at bay while designing real-time replication of data from operational RDBMSs, primarily due to the need for layering in multiple middlewares like CDCs, message brokers, message streams, etc. in different capacities which often result in a lot of plumbing.

Also, the commercially available middleware platforms are not always free of compatibility issues, integration complexities & high costs.

An alternate serverless option that I found handy is to leverage MySQL-compatible Amazon Aurora for such replications by leveraging its Stored Procedure that invokes Lambda functions.

While it needs data migration to Aurora it can enable an efficient and cost-effective event-driven architecture in combination with other AWS services of the serverless stack.

A common use case is a lambda function replicating changes from an Aurora table to S3, that are buffered through Kinesis Firehose, for analytical queries & visualizations. This lambda function will get invoked by an Aurora Stored Procedure, with “lambda” permissions, which in turn gets triggered by an “Insert” trigger defined on the Aurora table.

Here is a detailed blog post on this subject.

https://aws.amazon.com/blogs/database/capturing-data-changes-in-amazon-aurora-using-aws-lambda/

Suvo Dutta

I have over 22 years of IT experience in strategy, advisory, innovations, and cloud-based solutions in the Insurance domain. I advise clients in transforming their IT ecosystems to future-ready architectures that can provide exemplary customer experience, improve operating efficiency, enable faster product development and unlock the power of data.

You may also like...