Overview
MATILLION is an exciting technologies that has become prominent, and frequently used with databases like Snowflake and BigQuery. This blog will get you up to speed on some of the fundamental questions help you position it
What is the back story of Matillion?
- Dual headquarters in Denver and Manchester
- Offices in New York and Seattle
- Founded in 2011
- Founders Matthew Scullion
Is it Serverless?
- Matillion ETL is a PaaS (Platform as a Service). You need to spin up a server.
- Matillion Data Loader is SaaS a (Software as a Service) based data integration tool
How many products do they have?
- Matillion ETL for Amazon Redshift (available on the AWS Marketplace)
- Matillion ETL for Snowflake (available on the AWS and Microsoft Azure Marketplaces)
- Matillion ETL for Google BigQuery (available on the Google Cloud Marketplace)
- Matillion Data Loader
Can you use it on AWS, Azure and GCP?
Yes, see products above
Is it ETL or ELT?
It’s both, however, if you want to leverage the benefits of using BigQuery, SnowFlake, etc. You will adopt a ELT approach. The following is great article that discusses the pros and cons of the two approaches
Is it fail over aware?
In active-active high availability configuration if a node fails then the failure will be detected by another node and the job re-submitted from the start.
Is there a Graphical Development Environment
- Drag-and-drop browser interface.
- Live feedback, validation, and data preview.
- Enterprise features such as Generated Documentation and Data Lineage.
Does it integrate with GIT?
Yes, Git functionality lets users convert a Matillion project
into a Git project. This is limited to the Enterprise version of the product.
What is the licensing approach
There are currently 4 options
- Medium
- Large
- XLarge
- Enterprise
Which one is chosen will significantly impact cost and the number of developers that can work based on the concurrent users limitations. This is addressed by having multiple instances in the lower environments so that multiple teams can work in parallel.
Is there cap on what you can do?
Yes, there is a cap on the total number of environments that may exist within a installation. The cap depends upon the instance size.
How do I launch the browser based development environment?
This is as simple as launching a browser and using the external IP address provided by the Cloud Provider. The login details will be a generic user name (e.g. gcp-user) and password created during the instance created and shown by the Cloud Provider.
What is the overhead in maintaining the Compute Instance version?
There is a straight forward process in updating the version. Simply navigate to the Admin console and select Matillion ETL Updates
What is the difference between a Project Group and a Project Name
- Project Group: An existing or new Group of arbitrary name used for logically grouping Projects.
- Project Name: An arbitrary name for your new Project
Does Matillion security support LDAP integration?
Yes, there are two models for security, they are:
- Internal (Internal data store)
- External (LDAP)
How are jobs migrated between Dev, SIT, UAT, etc.
Options available to migrate code include
With Matillion Enterprise 1.40, GIT can be used as a mechanism for moving jobs between projects
The Export/Import feature for copying job metadata from one Instance (or Project) into another.
A REST API endpoints for exporting and importing Jobs are available
How does version control work?
There is native version control and GIT In Matillion ETL version 1.40, Enterprise customers. Matillion suggest that GIT is only used where manadated by the organisation using Matillion. This is due to the overhead of GIT processes
