feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources #7735
Conversation
github-actions
bot
added
the
ingestion
ANich
force-pushed
the
looker_lineage_timestamps
branch
2 times, most recently
from
e66adde to
1224d2e
Compare
| @@ -841,6 +843,10 @@ def _to_metadata_events( # noqa: C901 | |||
| UpstreamClass( | |||
| dataset=view_urn, | |||
| type=DatasetLineageTypeClass.VIEW, | |||
| auditStamp=AuditStamp( | |||
| time=int(observed_lineage_ts.timestamp() * 1000), | |||
| actor="urn:li:corpuser:datahub", | |||
There was a problem hiding this comment.
it'd be good to extract this out into a constant
| @@ -21,6 +21,7 @@ | |||
| ) | |||
|
|
|||
| import pydantic | |||
| from datahub.metadata.com.linkedin.pegasus2avro.common import AuditStamp | |||
There was a problem hiding this comment.
looks like linter doesn't like the import sorting here
|
@ANich - we haven't seen movement on this PR for a little while- are you still interested in contributing? If so, please let us know within the week, or we'll need to close this one for inactivity. |
anshbansal
added
the
community-contribution
Hey @laulpogan @anshbansal I'll pick this back up in the next couple days if that's okay. Will ping for a re-review |
ANich
force-pushed
the
looker_lineage_timestamps
branch
from
810923f to
2eb7aa2
Compare
ANich
force-pushed
the
looker_lineage_timestamps
branch
from
2eb7aa2 to
572ffd7
Compare
| @@ -786,6 +789,7 @@ def _to_metadata_events( # noqa: C901 | |||
| if self.upstream_views is not None: | |||
| assert self.project_name is not None | |||
| upstreams = [] | |||
| observed_lineage_ts = datetime.datetime.now() | |||
There was a problem hiding this comment.
| observed_lineage_ts = datetime.datetime.now() | |
| observed_lineage_ts = datetime.datetime.now(tz=timezone.utc) |
There was a problem hiding this comment.
Updated! Thanks for the review
| @@ -1615,11 +1617,16 @@ def _get_upstream_lineage( | |||
|
|
|||
| # Generate the upstream + fine grained lineage objects. | |||
| upstreams = [] | |||
| observed_lineage_ts = datetime.now() | |||
ANich
force-pushed
the
looker_lineage_timestamps
branch
from
572ffd7 to
137cb25
Compare
hsheth2
added
the
merge-pending-ci
|
cypress test failures unrelated so merging. |
Adds timestamps to the Looker and LookML sources for lineage between Explores and Views as well as between Views and SQL tables/views.
Checklist
- [ ] Links to related issues (if applicable)- [ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.- [ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub