Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ZEPPELIN-6027] Enhanced Integration with Apache Kyuubi #4770

Merged
merged 2 commits into from
Jul 3, 2024
Merged

[ZEPPELIN-6027] Enhanced Integration with Apache Kyuubi #4770

merged 2 commits into from
Jul 3, 2024

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Jun 28, 2024 •
edited
Loading

What is this PR for?

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on Lakehouses. Namely, the latest v1.9.1 adds support for Apache Spark 4.0.0-preview1, integrated Zeppelin with Kyuubi unblocks users to adopt new Spark seamlessly.

Kyuubi supports multiple computing engines, e.g. Spark, Flink, Hive, Trino, Doris, and multiple languages, e.g. SQL, Scala, PySpark.

This PR forces on the existing Spark SQL cases, adding support for:

  • extracting logs
  • jumping job web page

What type of PR is it?

Feature

What is the Jira issue?

ZEPPELIN-6027

How should this be tested?

Manually tested, as the existing Hive has no UT either.

Screenshots (if appropriate)

image

Xnip2024-06-28_22-33-44 Xnip2024-06-28_22-54-25 image

Questions:

  • Does the license files need to update? No.
  • Is there breaking changes for older versions? No,
  • Does this needs documentation? Yes, but maybe later, in dedicated PR.

pan3793
pan3793 commented Jun 28, 2024
View reviewed changes
jdbc/src/main/java/org/apache/zeppelin/jdbc/kyuubi/BeelineInPlaceUpdateStream.java

import org.apache.hadoop.hive.common.log.InPlaceUpdate;
import org.apache.hadoop.hive.common.log.ProgressMonitor;
import org.apache.kyuubi.jdbc.hive.logs.InPlaceUpdateStream;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kyuubi JDBC driver is derived from Hive JDBC driver, we relocated the classes and striped all 3rd party dependencies.

pan3793
pan3793 commented Jun 28, 2024
View reviewed changes
jdbc/src/main/java/org/apache/zeppelin/jdbc/kyuubi/KyuubiUtils.java

if (!jobUrlExtracted) {
String appId = kyuubiConn.getEngineId();
String appUrl = kyuubiConn.getEngineUrl();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the enhancement in Kyuubi JDBC, so we don't need to extract engine app URL from logs.

pan3793
pan3793 commented Jun 28, 2024
View reviewed changes
jdbc/src/main/java/org/apache/zeppelin/jdbc/kyuubi/KyuubiUtils.java
String appId = kyuubiConn.getEngineId();
String appUrl = kyuubiConn.getEngineUrl();
String jobUrl = null;
// prefer to use customized template, and fallback to Kyuubi returned engine url
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in case the Hadoop cluster is deployed behind Gateway, e.g. Knox, the users at the corp zone may need to use a different URL to access the Application Web page

@pan3793
Copy link
Member Author

pan3793 commented Jun 28, 2024

cc @zjffdu, could you please take a look?

@pan3793
Copy link
Member Author

pan3793 commented Jun 28, 2024

also cc @huage1994

@zjffdu
Copy link
Contributor

zjffdu commented Jun 28, 2024

Thanks for the contribution, @pan3793 LGTM

huage1994
huage1994 approved these changes Jul 1, 2024
View reviewed changes
Copy link
Contributor

@huage1994 huage1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pan3793
Copy link
Member Author

pan3793 commented Jul 3, 2024

@zjffdu @huage1994 thanks for your approval, can we get this in?

@zjffdu zjffdu merged commit 1980817 into apache:master Jul 3, 2024
26 of 28 checks passed
JeffreySmith added a commit to acceldata-io/zeppelin that referenced this pull request Aug 7, 2024
@JeffreySmith @bigpear0201
@dependabot @shefali163 @jongyoul @Reamer @pjfanning @myongyun @th1nksnow @kevinjmh @pan3793 @flowy0 @zeotuan @aditi-sharma-1 @prabhjyotsingh @shubhluck
ODP-1829: Update Zeppelin to 11.2 for JDK11 (#6)
5b508a3 
* [MINOR] Update jdbc.md Change postgres username (apache#4704)

change postgres default username mysql_user/mysql_password to pg_user/pg_password.

* Bump mathjax from 2.7.0 to 3.0.0 in /zeppelin-web (apache#4705)

Bumps [mathjax](https://github.com/mathjax/MathJax) from 2.7.0 to 3.0.0.
- [Release notes](https://github.com/mathjax/MathJax/releases)
- [Commits](mathjax/MathJax@2.7.0...3.0.0)

---
updated-dependencies:
- dependency-name: mathjax
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump org.apache.shiro:shiro-core from 1.10.0 to 1.13.0 (apache#4703)

Bumps [org.apache.shiro:shiro-core](https://github.com/apache/shiro) from 1.10.0 to 1.13.0.
- [Release notes](https://github.com/apache/shiro/releases)
- [Changelog](https://github.com/apache/shiro/blob/main/RELEASE-NOTES)
- [Commits](apache/shiro@shiro-root-1.10.0...shiro-root-1.13.0)

---
updated-dependencies:
- dependency-name: org.apache.shiro:shiro-core
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump follow-redirects from 1.15.3 to 1.15.4 in /zeppelin-web (apache#4702)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](follow-redirects/follow-redirects@v1.15.3...v1.15.4)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump follow-redirects from 1.15.3 to 1.15.4 in /zeppelin-web-angular (apache#4701)

Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](follow-redirects/follow-redirects@v1.15.3...v1.15.4)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [MINOR] Upgrade jackson version in /zeppelin-server (apache#4635)

* [HOTFIX] Disable shell interpreter (apache#4708)

* Fix GCSNotebookRepoTests (apache#4711)

* [HOTFIX] Check permission when updating cron information (apache#4631)

* [HOTFIX] Check permission when updating cron information

* [HOTFIX] Fix commented

* [HOTFIX] Check permission when updating cron information

* [HOTFIX] Check permission when updating cron information

* [HOTFIX] Check permission when updating cron information

* [MINOR] Change minimum java version to 11 in docs (apache#4710)

* [MINOR] Change minimum java version to 11 in docs

* [MINOR] Change minimum java version to 11 in docs

* [HOTFIX] Change the link of `helium.json` from S3 to zeppelin.apache.org (apache#4713)

* [ZEPPELIN-5990] Disable sensitive configuration for JDBC url (apache#4709)

* [ZEPPELIN-5990] Disable sensitive configuration for JDBC url

* [ZEPPELIN-5990] Disable sensitive configuration for JDBC url

* [ZEPPELIN-5995] Update Kubernetes Library and hopefully fix flaky tests (apache#4712)

* [MINOR] Set Snapshot version to 0.12.0-SNAPSHOT (apache#4720)

* change version to 0.11.1-SNAPSHOT

* change version

* change to 0.12.0-SNAPSHOT

* Bump org.postgresql:postgresql from 42.4.3 to 42.7.2 in /jdbc (apache#4723)

Bumps [org.postgresql:postgresql](https://github.com/pgjdbc/pgjdbc) from 42.4.3 to 42.7.2.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pgjdbc/pgjdbc/commits)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [HOTFIX] Escape Ldap search filters (apache#4714)

* [HOTFIX] Escape envs when using `.conf` (apache#4715)

* Fix the new zeppelin ui. The specific reason is that the use of excessively wide column widths resulted in the remaining fields being squeezed out of the screen, and nz table did not have a scroll bar set to display scrolling. (apache#4727)

* Bump ip from 1.1.8 to 1.1.9 in /zeppelin-web (apache#4724)

Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9.
- [Commits](indutny/node-ip@v1.1.8...v1.1.9)

---
updated-dependencies:
- dependency-name: ip
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump lodash from 4.17.15 to 4.17.21 in /zeppelin-web-angular (apache#4689)

Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.15...4.17.21)

---
updated-dependencies:
- dependency-name: lodash
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [HOTFIX] Escape HeliumPackage information (apache#4728)

* [NO-ISSUE] Use reload4j instead of log4j (apache#4719)

* [NO-ISSUE] Upgrade org.json jar (apache#4722)

* [ZEPPELIN-6001] k8s images fix (apache#4729)

* Update zeppelin-interpreter Dockerfile

fixed ARG version;
Error "bzip2: Compressed file ends unexpectedly;" fixed by adding ---no-iri flag to wget

* Update zeppelin-server Dockerfile

fixed ARG version;
fixed JAVA_HOME env.

* Update scripts/docker/zeppelin-interpreter/Dockerfile

Co-authored-by: Philipp Dallig <[email protected]>

---------

Co-authored-by: Philipp Dallig <[email protected]>

* [ZEPPELIN-6000] Polish some files mainly in zengine (apache#4731)

* some misc polish

* some misc polish

* [ZEPPELIN-6003] Log source info of SQL in JDBCInterpreter (apache#4732)

* [ZEPPELIN-6003] Log detail info of SQL in JDBCInterpreter

* Update

Co-authored-by: Philipp Dallig <[email protected]>

---------

Co-authored-by: Philipp Dallig <[email protected]>

* [ZEPPELIN-6002] Fix completer NPE (apache#4730)

Co-authored-by: Philipp Dallig <[email protected]>

* [ZEPPELIN-5986] Bump Maven surefire/failsafe plugins to recover JUnit5 tests (apache#4734)

* [ZEPPELIN-5986] Re-enable Junit 5 integration tests by upgrading maven plugins

* Fix SparkIntegrationTest

* Disable the LivyInterpreterIT

* Selenium

* NPE

* Disable testEditOnDoubleClick

* [ZEPPELIN-6005] Update Kyuubi JDBC docs (apache#4738)

* Bump org.apache.commons:commons-configuration2 from 2.8.0 to 2.10.1 (apache#4740)

Bumps org.apache.commons:commons-configuration2 from 2.8.0 to 2.10.1.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-configuration2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [MINOR] Fix missing changed versions (apache#4737)

* [MINOR] Fix missing changed versions

* nit

* Bump com.nimbusds:nimbus-jose-jwt in /zeppelin-server (apache#4733)

Bumps [com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.13 to 9.37.2.
- [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt)
- [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/9.37.2..9.13)

---
updated-dependencies:
- dependency-name: com.nimbusds:nimbus-jose-jwt
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [ZEPPELIN-6007] Enhance release scripts for tar shasum commands detection (apache#4747)

* [ZEPPELIN-6007] Enhance release scripts for tar shasum commands detection

* fix

* nit

* indent

* [ZEPPELIN-6008] Fix parameter usage of bokeh in test case (apache#4748)

* [ZEPPELIN-6008] Pin plotly 5.19.0

* Revert "[ZEPPELIN-6008] Pin plotly 5.19.0"

This reverts commit 60d9ce9.

* Pin bokeh=3.3.4

* Revert "Pin bokeh=3.3.4"

This reverts commit 44ccc98.

* Remove usage of deprecated paramater legend

* [ZEPPELIN-5969] Remove Hadoop2 and move to Hadoop3 shaded client (apache#4691)

* Drop hadoop2 in github actions

* Update docs

* Drop hadoop2 support

* Remove hadoop2 integration tests

* findbugs use the same version in all modules

* Use hadoop3.3 for tests

* Move to scala 2.12

* Try to fix flink

* Usage of metals

* Remove duplicate version and groupid

* Fix Flink with Hadoop3

* fix log

* R

* fix

* fix

* fix

* fix

* hadoop-3.3

* fix

* fix

* Address comments

* address comments

---------

Co-authored-by: Philipp Dallig <[email protected]>

* [HOTFIX] Remove rendering helium description as HTML in Frontend (apache#4755)

* Bump express from 4.18.2 to 4.19.2 in /zeppelin-web-angular (apache#4744)

Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](expressjs/express@4.18.2...4.19.2)

---
updated-dependencies:
- dependency-name: express
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* update scripts/docker/zeppelin/bin/Dockerfile to JDK11 (apache#4753)

* update scripts/docker/zeppelin/bin/Dockerfile to JDK11

* update miniconda to py3.9

* Update Dockerfile to remove extra line

* Bump some apache common libs (apache#4757)

* [ZEPPELIN-6006] Remove command line applications when downloading applications (apache#4746)

* Move Files with java

* Use java to download external dependecies

* Improve code after review

* Correct Mirror-URL and compilation

* [ZEPPELIN-6016] Rewrite and enable Livy integration tests (apache#4743)

* wip

* nit

* nit

* wip

* wip

* fix

* [ZEPPELIN-5973] Bump Livy 0.8.0-incubating

* nit

* Spark 3.5.1

* test

* fix

* comment

* nit

* nit

* nit

* [ZEPPELIN-6017] Revert changes about ZEPPELIN_IDENT_STRING in ZEPPELIN-5421

* [ZEPPELIN-6015] Update ci-action plugins (apache#4759)

* [ZEPPELIN-5999] Reduce instance objects from Zeppelin (apache#4726)

* Remove ZeppelinConfiguration Singelton and add MiniZeppelinServer

* Add ZeppelinConfiguration to Interpreter

* Remove static pluginmanager and configstorage

* Inject servicelocator into SessionConfiguratior

* use custom serviceLocator in integration tests

* Reorder code

* code cleanup

* Add ZeppelinConfiguration as class variable to InterpreterOption

* Avoid leaking third-party libs

* [ZEPPELIN-6019] Remove Submarine support (apache#4763)

* [ZEPPELIN-6022] Skip decryption of credentials.json when file is empty (apache#4765)

* Skip decryption when empty

* Use more elegant empty json string check

* [ZEPPELIN-6018] Update gRPC version from 1.51.0 to 1.55.1 for successful Apache Zeppelin build on s390x architecture (apache#4758)

* Update grpc version in pom.xml for successful Apache Zeppelin build on s390x architecture

In the s390x architecture, the Apache Zeppelin package builds successfully when the grpc dependency version in the pom.xml file is changed from version 1.51.0 to version 1.62.2. Therefore, I have updated the grpc version in the pom.xml file.

* Update LICENSE

Updated the grpc version in LICENSE from 1.51.0 to 1.62.2

* grpc-version-change-to 1.55.1

* [ZEPPELIN-6027] Enhanced Integration with Apache Kyuubi (apache#4770)

* [ZEPPELIN-6027] Enhanced Integration with Apache Kyuubi

* fix style

* [ZEPPELIN-6028] Enhance default value assignment for ZEPPELIN_IDENT_STRING (apache#4772)

* [ZEPPELIN-6029] Set COPYFILE_DISABLE=1 for macOS tar (apache#4774)

* [ZEPPELIN-6029] Add --disable-copyfile for macOS tar

* COPYFILE_DISABLE=1

* ODP-1644: Removed unsupported interpreter

* ODP-1315: Bumpup loadash to 4.x

* ODP: fix odp version

* ODP-303 New UI build fix for missing projects

* [ODP-1315] ODP-1644: CVE fixes

* ODP-1644: Update to Spark Version

* ODP-1644: Added libthrift, updated hadoop and phoenix versions

* ODP-1644: Hive 4.0.0 support for JDBC

* ODP-1644: Added two variables that were previously missed

* ODP-1644: Fixed node/npm version issue when testing with arm64 MacOS

* ODP-1644: Increased timeouts/sleeps to pass tests

* ODP-1644: Increased ms in Thread.sleep

* ODP-1829: Updated pom.xml files

* ODP-1829: Removed uneeded tests

* ODP-1829: Fixed tests failures

* ODP-1829: Disabled Finicky Helium Test

* ODP-1829: Increased wait time so that tests would not fail

* ODP-1829: Thread.Sleep -> Thread.sleep

* Zeppelin Build fixes

* ODP-1644: Updated version number to 11.2.3.3.6.0-1

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: bigpear0201 <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shefali Singh <[email protected]>
Co-authored-by: Jongyoul Lee <[email protected]>
Co-authored-by: Philipp Dallig <[email protected]>
Co-authored-by: PJ Fanning <[email protected]>
Co-authored-by: myongyun <[email protected]>
Co-authored-by: th1nksnow <[email protected]>
Co-authored-by: Manhua <[email protected]>
Co-authored-by: Manhua <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Co-authored-by: Gayle <[email protected]>
Co-authored-by: zeotuan <[email protected]>
Co-authored-by: Aditi Sharma <[email protected]>
Co-authored-by: Prabhjyot Singh <[email protected]>
Co-authored-by: shubhamsharma <[email protected]>
@pan3793 pan3793 deleted the ZEPPELIN-6027 branch September 20, 2024 05:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@huage1994 huage1994 huage1994 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pan3793 @zjffdu @huage1994