About Download Matillion
Matillion software deployment scenario we are following, that means: The data engineers and data scientists only commit artifacts that they have changedThe DevOps pipeline runs for every commitIn a Matillion ETL context, the DevOps pipeline deliberately targets only the changes that involve the Matillion ETL codebaseGathering and committing the changed artifacts is done using the Matillion ETL REST API, as we saw in the previous section. With Azure DevOps you can easily make a pipeline run upon every commit. You choose the name of the GitHub event that should trigger the workflow by setting the "on" element in the root map. In our worked example, it looks like this:name: Build and Deploy Matillion ETL appon: push: branches: - masterMatillion's REST API generates JSON files for the job definitions. Within the workflow, it's not difficult to isolate the changes by checking out the entire repository into the runner, and then using a git metadata query. Our implementation does as follows: # Checkout the repo so the workflow can access it - name: Sync repository uses: actions/checkout@v2 with: fetch-depth: 0 # List the changed files - name: List changed run: | git diff --diff-filter=AM --name-only ${{ github.event.before }}..${{ github.event.after }}A section of the event stream from a full deployment run is shown below. It only contains a single Matillion ETL job: In summary, the DevOps workflow has determined that only one Matillion job was changed, so that's the only one that needs to be replaced on the target Matillion ETL instance. This last best practice focuses on making sure the DevOps process only involves aspects of the solution that have been changed. This approach enables you to start small and scale later as the project requires. It's highly targeted and quick to run. This is a vital enabler for the early and continuous delivery of valuable software. Next stepsThis article has focused on best practices for managing Azure DevOps CI/CD Pipelines using one specific technique with Matillion ETL. As next steps, you might consider taking advantage of more sophisticated features such as:ApprovalsAdding more Azure service deployments, for example to integrate with Azure Queue Storage and the corresponding Matillion ETL functionalityAdding pipelines to your Azure Boards as part of agile project managementIntroducing Exploratory & Manual TestingVisiting the CI/CD solution ideas area in the Azure documentationDownloadsFor reference, you can download the full GitHub workflow script that we have been using.Integrate Matillion ETL with the
Key features of the Download Matillion that are enabled through these drivers include:
- How to? Orchestration - Matillion Academy
- Using Stored Procedures with Matillion
- Connect to Microsoft Sharepoint - Matillion
- Matillion recognized again as a Challenger by the
- Introducing Matillion ETL for Snowflake
- A Guide For Data Engineers - Matillion
- How To: Download Matillion ETL Server Log in the UI
Of standard libraries installed. Additionally, and frequently, Matillion ETL users may want to install additional Python libraries on their Matillion ETL instances. This can be done using pip, Python's package manager. Read Additional Modules to learn how to add Python libraries to a Matillion ETL instance.As of version 1.61.6, Matillion ETL comes with version 8.1.2 of pip for Python2 and version 9.0.3 of pip for Python3. Typically, the version of pip that ships with Matillion ETL should be sufficient for most use cases and we do not recommend updating the version of pip. Matillion ETL releases are tested based on the version of pip that is included with Matillion ETL. Installing a different version of pip could result in unexpected behavior within Matillion ETL. If you find a need to upgrade the version of pip on your Matillion ETL instance, please ensure you have a current backup of your instance that can be restored if necessary.Migrating Matillion InstancesThere can be many reasons to launch a new Matillion ETL instance and migrate your existing Matillion ETL instance over to it. The most common scenario for this is when updating to a new version of Matillion ETL following our recommended best practice to launch a new Matillion instance and then use Matillion's Migrate feature. When migrating, note that Python libraries are included in the list of resources that are not automatically migrated. We will outline here the best practice steps for migrating Python libraries that your Matillion ETL jobs may have dependencies on, to a new Matillion ETL instance.Confirm versions of Python and Pip on the CURRENT Matillion ETL instance.SSH on to the Matillion instanceRun the following commands and note the output:Current version of Python2 installed: sudo python --versionCurrent version of Pip installed for Python2: sudo pip --versionCurrent version of Python3 installed: sudo python3 --versionCurrent version of Pip installed for Python3: sudo pip3 --versionConfirm the Python Libraries installed and the version of each on the CURRENT Matillion ETL instance.SSH on to the Matillion instanceRun the following commands and note the output:List of installed Python2 libraries: sudo pip freezeList of installed Python3 libraries:
