Daton Comparison

Daton is an enterprise grade ETL tool that is trusted by 100s of customers and is proven at scale. Daton is designed to be an eCommerce and Retail focused data pipeline that will fulfill your current and future replication needs. What’s more? Daton support and consulting teams are always just a message away from supporting you get the best out of your investment.

What is ETL?

 ETL is a process involving extraction (E) of data from disparate sources, transformation (T) of data in a staging server by performing actions such as modifying the data types or application of specific calculations, and then loading (T) data into the target location – usually a data warehouse.

ETL processes are typically associated with relational databases. Any structured or unstructured data should be transformed into a relational form before loading into a data warehouse. A staging server that usually runs independently of the data warehouse executes the transformation logic, which may include data filtering, sensitive data masking, data enrichment, data mapping, data de-duplication, and combining data from multiple sources.

Because transformations are happening before data is loaded, the data structures required to load the data must be pre-defined. Therefore, careful planning and preparation must go into this exercise to determine the right data structures to support business requirements. Any lapses in this phase may result in the entire data pipeline rework, leading to delays and cost escalations.

ETL has been the dominant strategy for data movement for many decades. However, that approach is being challenged now with the rise of cloud data warehouses. Cloud data warehouses make it less expensive and easy to store and process data. This change had led to the application of new  ETL techniques to build the data warehouse, where experimentation and rapid iteration have replaced careful planning and execution.

Let’s look at the important criteria that ETL tools should support to make the life of an analyst or a data engineer easy rather than becoming a burden.

Number of Connectors

Multiple applications are used to operate a business and it is important for any ETL tool to support many applications so that customers can rely, as much as possible, a single ETL tool for all their replication needs.

Destination Support

Customers have a lot of choice when it comes to which data warehouse they want to use for their reporting and analytics. Therefore, it is important for any ETL tool to support not just the cloud data warehouses like Snowflake and BigQuery, but also databases like MySQL, PostgreSQL and cloud data storages like Amazon S3 if customers want to take the data lake approach.

Loading Controls

Most cloud warehouses charge customers for the amount and frequency of data loaded into the data warehouse. Therefore, customers should seek solutions that give them as much control as possible while loading data. How ETL tools handle nested data, how they handle updates to data at source are a couple of important factors that determine how much time analysts spend once the ETL tool lands data in the data warehouse.

Replication Scheduling

In addition to loading controls, having control on how frequently data is loaded into the data warehouse is also an important criterion. If you take Snowflake as an example, Snowflake charges customers for the compute time utilized to load data. Often, customers need replication jobs to run a certain time in the day (I want my jobs to run at 8am everyday) for which CRON expression-based scheduling is needed.

Many ETL tools provide flexible scheduling options, but most tools offer scheduling at an integration level where all the tables within an integration inherit the same schedule. However, business requirements may demand a more flexible scheduling routines where some tables in the integration replicate data faster and some tables can be replicated at a slower frequency. When table level schedules are not supported, customers must create and manage multiple integrations which can not only be tedious but may also consume their available quota for integrations under their current pricing plan.

Pipeline Visibility

By choosing an ETL tool, customers are handing over the responsibility of replicating data to the ETL tool provider. Therefore, it is imperative that the ETL tool provides customers visibility into its functioning so that customers have peace of mind knowing that the tool is doing what it promises to achieve. If customers don’t have visibility into the pipeline or are not able to replicate the job data to their data warehouse, then they are totally reliant on the ETL tool user interface for any troubleshooting activity. Analyst who is comfortable with SQL or Python, typically prefers to monitor the pipeline by using code and to achieve this, the ETL tool should be able to synchronize job data to a destination of customer’s choice.

Webhooks

Not all sources can be covered by a single ETL tool. However, by using generic Webhooks functionality, customers can replicate data from any source that support Webhooks. This is a complex feature that can only be achieved by advanced ETL tools that support automated schema management.

Source Groups

For customers that have 100s of integrations, it can be quite useful to group their source integrations into defined groups. This will not only help them easily navigate through the UI of the ETL tool, but it will also makes it easy to assess their costs at each group level

Source Templates

Customers, especially B2B customers, have needs to create 10s of integrations for the same source. Managing the settings for these sources and achieving consistency in the schemas becomes critical to ensure changes to the configuration don’t break downstream pipelines.

Price

Pricing is an important factor to consider while selecting an ETL tool. Pricing should be easy to calculate, transparent, and controllable.