PGQ excels at asynchronous task processing and offers deep observability. This makes it a key component of the Dataddo platform.
Hi. We are Tomas and Tomas, CTO and Backend Team Lead at Dataddo, a company that provides a fully managed, end-to-end data integration platform. Clients in time zones across the globe use our platform, so, as you can imagine, we need to process loads of integration jobs at several peak periods throughout the day.
Optimized queuing, therefore, is essential for preventing the memory bloating of our data transferring and processing services, and ensuring the stability of our platform.
But finding a tool that could meet all our queueing needs proved to be a challenge. So, we built our own: PGQ—a free, open-source message broker built on top of Postgres, which is specifically designed to handle long-running jobs.
Before building PGQ, we considered a range of queueing tools and eventually decided that RabbitMQ would be the most suitable. RabbitMQ is indeed excellent for short-running jobs, but it couldn’t efficiently meet all our needs because:
PGQ solves all of these issues for us because:
We use PGQ for four main things:
PGQ could work well for you if:
One example of an organization for which PGQ might be suitable: An ecommerce company whose eshop gets thousands of orders per day, which already has Postgres in place, and which doesn’t want to deploy a big-guns solution like Kafka.
It’s also worth mentioning that PGQ is currently only for Go. However, we already use PHP consumers and publishers internally, and will soon release a PHP open-source package to the public.
PGQ is probably not for you if:
Every day, the Dataddo platform processes thousands of large and small extract/write jobs—ELT/ELT jobs, reverse ETL jobs, database replication jobs, and event-based integrations. Many of these jobs are very long-running (for example, due to underperforming customer databases, or huge data loads in case of full database replication).
Other message brokers either can’t handle long-running jobs very well (e.g., RabbitMQ), or are way overcomplicated for our use case (e.g., Apache Kafka). PGQ handles long-running jobs perfectly, while still being very simple in design. This is why we want to share it with the developer community.
Think PGQ might be suitable for you? See PGQ details.
Connect All Your Data with Dataddo Move your data from any online services to any data warehouse, between any two warehouses, and from any warehouse into any operational applications. |