Merge branch 'datahub-project:master' into looker_lineage_timestamps

docs-website/sidebars.js

@@ -71,6 +71,13 @@ module.exports = {
                 "docs/quick-ingestion-guides/bigquery/configuration",
               ],
             },
+            {
+              Redshift: [
+                "docs/quick-ingestion-guides/redshift/overview",
+                "docs/quick-ingestion-guides/redshift/setup",
+                "docs/quick-ingestion-guides/redshift/configuration",
+              ],
+            },
             {
               Snowflake: [
                 "docs/quick-ingestion-guides/snowflake/overview",

docs/quick-ingestion-guides/bigquery/setup.md

@@ -3,7 +3,7 @@ title: Setup
 ---
 # BigQuery Ingestion Guide: Setup & Prerequisites
 
-To configure ingestion from BigQuery, you'll need a [Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) configured with the proper permission sets, and an associated [Service Account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).
+To configure ingestion from BigQuery, you'll need a [Service Account](https://cloud.google.com/iam/docs/creating-managing-service-accounts) configured with the proper permission sets and an associated [Service Account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).
 
 This setup guide will walk you through the steps you'll need to take via your Google Cloud Console.
 
@@ -27,11 +27,12 @@ Please refer to the BigQuery [Permissions](https://cloud.google.com/iam/docs/per
 1. To set up a new Service Account follow [this guide](https://cloud.google.com/iam/docs/creating-managing-service-accounts)
 
 2. When you are creating a Service Account, assign the following predefined Roles:
-   - [BigQuery Job User](https://cloud.google.com/bigquery/docs/access-control#bigquery.jobUser)
-   - [BigQuery Metadata Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer)
-   - [BigQuery Resource Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.resourceViewer) -> This role is for Table-Level Lineage and Usage extraction
-   - [Logs View Accessor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer) -> This role is for Table-Level Lineage and Usage extraction
-   - [BigQuery Data Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer) -> This role is for Profiling
+   * [BigQuery Job User](https://cloud.google.com/bigquery/docs/access-control#bigquery.jobUser)
+   * [BigQuery Metadata Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer)
+   * [BigQuery Resource Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.resourceViewer) -> This role is for Table-Level Lineage and Usage extraction
+   * [Logs View Accessor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer) -> This role is for Table-Level Lineage and Usage extraction
+   * [BigQuery Data Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer) -> This role is for Profiling
+   * [BigQuery Read Session User](https://cloud.google.com/bigquery/docs/access-control#bigquery.readSessionUser) -> This role is for Profiling
 
 :::note
 You can always add/remove roles to Service Accounts later on. Please refer to the BigQuery [Manage access to projects, folders, and organizations](https://cloud.google.com/iam/docs/granting-changing-revoking-access) guide for more details.
@@ -40,6 +41,7 @@ You can always add/remove roles to Service Accounts later on. Please refer to th
 3. Create and download a [Service Account Key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys). We will use this to set up authentication within DataHub.
 
 The key file looks like this:
+
 ```json
 {
    "type": "service_account",
@@ -54,7 +56,9 @@ The key file looks like this:
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/test%suppproject-id-1234567.iam.gserviceaccount.com"
 }
 ```
+
 ## Next Steps
+
 Once you've confirmed all of the above in BigQuery, it's time to [move on](configuration.md) to configure the actual ingestion source within the DataHub UI.
 
 *Need more help? Join the conversation in [Slack](http://slack.datahubproject.io)!*

docs/quick-ingestion-guides/redshift/configuration.md

@@ -0,0 +1,136 @@
+---
+title: Configuration
+---
+# Configuring Your Redshift Connector to DataHub
+
+Now that you have created a DataHub user in Redshift in [the prior step](setup.md), it's time to set up a connection via the DataHub UI.
+
+## Configure Secrets
+
+1. Within DataHub, navigate to the **Ingestion** tab in the top, right corner of your screen
+
+<p align="center">
+  <img width="75%" alt="Navigate to the &quot;Ingestion Tab&quot;" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_ingestion_button.png"/>
+</p>
+
+:::note
+If you do not see the Ingestion tab, please contact your DataHub admin to grant you the correct permissions
+:::
+
+2. Navigate to the **Secrets** tab and click **Create new secret**
+
+<p align="center">
+   <img width="75%" alt="Secrets Tab" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_secrets_tab.png"/>
+</p>
+
+3. Create a Redshift User's Password secret
+
+This will securely store your Redshift User's password within DataHub
+
+* Click **Create new secret** again
+* Enter a name like `REDSHIFT_PASSWORD` - we will use this later to refer to the secret
+* Enter your `datahub` redshift user's password
+* Optionally add a description
+* Click **Create**
+
+<p align="center">
+   <img width="75%" alt="Redshift Password Secret" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_password_secret.png"/>
+</p>
+
+## Configure Recipe
+
+4. Navigate to the **Sources** tab and click **Create new source**
+
+<p align="center">
+  <img width="75%" alt="Click &quot;Create new source&quot;" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_click_create_new_source_button.png"/>
+</p>
+
+5. Select Redshift
+
+<p align="center">
+  <img width="75%" alt="Select BigQuery from the options" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_redshift_button.png"/>
+</p>
+
+6. Fill out the Redshift Recipe
+
+Populate the Password field by selecting Redshift Password secrets you created in steps 3 and 4.
+
+<p align="center">
+  <img width="75%" alt="Fill out the Redshift Recipe" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift-ingestion-recipe.png"/>
+</p>
+
+<!---
+
+7. Click **Test Connection**
+
+This step will ensure you have configured your credentials accurately and confirm you have the required permissions to extract all relevant metadata.
+
+<p align="center">
+  <img width="75%" alt="Test BigQuery connection" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/bigquery/bigquery-test-connection.png"/>
+</p>
+
+After you have successfully tested your connection, click **Next**.
+
+-->
+
+## Schedule Execution
+
+Now it's time to schedule a recurring ingestion pipeline to regularly extract metadata from your Redshift instance.
+
+7. Decide how regularly you want this ingestion to run-- day, month, year, hour, minute, etc. Select from the dropdown
+
+<p align="center">
+    <img width="75%" alt="schedule selector" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_scheduled_execution.png"/>
+</p>  
+
+8. Ensure you've configured your correct timezone
+
+<p align="center">
+    <img width="75%" alt="timezone_selector" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_timezone_selector.png"/>
+</p>  
+
+9. Click **Next** when you are done
+
+## Finish Up
+
+10. Name your ingestion source, then click **Save and Run**
+
+<p align="center">
+  <img width="75%" alt="Name your ingestion" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_name_ingestion.png"/>
+</p>  
+
+You will now find your new ingestion source running
+
+<p align="center">
+  <img width="75%" alt="ingestion_running" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_running.png"/>
+</p>  
+
+## Validate Ingestion Runs
+
+11. View the latest status of ingestion runs on the Ingestion page
+
+<p align="center">
+  <img width="75%" alt="ingestion succeeded" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_succeeded.png"/>
+</p>  
+
+12. Click the plus sign to expand the full list of historical runs and outcomes; click **Details** to see the outcomes of a specific run
+
+<p align="center">
+  <img width="75%" alt="ingestion_details" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_details.png"/>
+</p>
+
+13. From the Ingestion Run Details page, pick **View All** to see which entities were ingested
+
+<p align="center">
+  <img width="75%" alt="ingestion_details_view_all" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_details_view_all.png"/>
+</p>  
+
+14. Pick an entity from the list to manually validate if it contains the detail you expected  
+
+<p align="center">
+  <img width="75%" alt="ingestion_details_view_all" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/redshift/redshift_ingestion_ingested_assets.png"/>
+</p>  
+
+**Congratulations!** You've successfully set up Redshift as an ingestion source for DataHub!
+
+*Need more help? Join the conversation in [Slack](http://slack.datahubproject.io)!*

docs/quick-ingestion-guides/redshift/overview.md

@@ -0,0 +1,38 @@
+---
+title: Overview
+---
+# Redshift Ingestion Guide: Overview
+
+## What You Will Get Out of This Guide
+
+This guide will help you set up the Redshift connector through the DataHub UI to begin ingesting metadata into DataHub.
+
+Upon completing this guide, you will have a recurring ingestion pipeline that will extract metadata from Redshift and load it into DataHub. This will include to following Redshift asset types:
+
+* Database
+* Schemas (External and Internal)
+* Tables (External and Internal)
+* Views
+
+This recurring ingestion pipeline will also extract:
+
+* **Usage statistics** to help you understand recent query activity
+* **Table-level lineage** (where available) to automatically define interdependencies between datasets
+* **Table- and column-level profile statistics** to help you understand the shape of the data
+
+:::caution
+The source currently can ingest one database with one recipe
+:::
+
+## Next Steps
+
+If that all sounds like what you're looking for, navigate to the [next page](setup.md), where we'll talk about prerequisites
+
+## Advanced Guides and Reference
+
+If you're looking to do something more in-depth, want to use CLI instead of the DataHub UI, or just need to look at the reference documentation for this connector, use these links:
+
+* Learn about CLI Ingestion in the [Introduction to Metadata Ingestion](../../../metadata-ingestion/README.md)
+* [Redshift Ingestion Reference Guide](https://datahubproject.io/docs/generated/ingestion/sources/redshift/#module-redshift)
+
+*Need more help? Join the conversation in [Slack](http://slack.datahubproject.io)!*

docs/quick-ingestion-guides/redshift/setup.md

@@ -0,0 +1,35 @@
+---
+title: Setup
+---
+# Redshift Ingestion Guide: Setup & Prerequisites
+
+To configure ingestion from Redshift, you'll need a [User](https://docs.aws.amazon.com/redshift/latest/gsg/t_adding_redshift_user_cmd.html) configured with the proper permission sets, and an associated.
+
+This setup guide will walk you through the steps you'll need to take via your Google Cloud Console.
+
+## Redshift Prerequisites
+
+1. Connect to your Amazon Redshift cluster using an SQL client such as SQL Workbench/J or Amazon Redshift Query Editor with your Admin user.
+2. Create a [Redshift User](https://docs.aws.amazon.com/redshift/latest/gsg/t_adding_redshift_user_cmd.html) that will be used to perform the metadata extraction if you don't have one already.
+For example:
+
+```sql
+CREATE USER datahub WITH PASSWORD 'Datahub1234';
+```
+
+## Redshift Setup
+
+1. Grant the following permission to your `datahub` user:
+
+```sql
+ALTER USER datahub WITH SYSLOG ACCESS UNRESTRICTED;
+GRANT SELECT ON pg_catalog.svv_table_info to datahub;
+GRANT SELECT ON pg_catalog.svl_user_info to datahub;
+
+```
+
+## Next Steps
+
+Once you've confirmed all of the above in Redshift, it's time to [move on](configuration.md) to configure the actual ingestion source within the DataHub UI.
+
+*Need more help? Join the conversation in [Slack](http://slack.datahubproject.io)!*

docs/quick-ingestion-guides/snowflake/setup.md

@@ -52,6 +52,9 @@ In order to configure ingestion from Snowflake, you'll first have to ensure you
    grant references on all views in database identifier($db_var) to role datahub_role;
    grant references on future views in database identifier($db_var) to role datahub_role;
 
+   -- Assign privileges to extract lineage and usage statistics from Snowflake by executing the below query.
+   grant imported privileges on database snowflake to role datahub_role;
+
    ```
 
    If you have imported databases in your Snowflake instance that you wish to integrate with DataHub, you'll need to use the below query for them.
@@ -60,14 +63,8 @@ In order to configure ingestion from Snowflake, you'll first have to ensure you
    grant IMPORTED PRIVILEGES on database "<your-database>" to role datahub_role;  
    ```
 
-4. Assign privileges to extract lineage and usage statistics from Snowflake by executing the below query.
-
-   ```sql
-   grant imported privileges on database snowflake to role datahub_role;
-   ```
-
 ## Next Steps
 
 Once you've done all of the above in Snowflake, it's time to [move on](configuration.md) to configuring the actual ingestion source within DataHub.
 
-*Need more help? Join the conversation in [Slack](http://slack.datahubproject.io)!*
+*Need more help? Join the conversation in [Slack](http://slack.datahubproject.io)!*