This post takes you through setting up the Databricks Connect client on windows and is a requirement for subsequently working with a Visual Studio environment.
The case of this example, the cluster is running Databricks 6.5 so all commands will reflect this. I have assumed that you are able to create a Workspace, this is a prerequisite.
Create and Start a Databricks Cluster
For the sake of this example I have create the simplest, single node cluster possible, without auto scaling
Once it is created and running, select the cluster and copy the URL:
The key information to note is highlighted above and translates as follows:
| Organisation Id | 1234567891011121314 |
| Cluster Id | 0621-093110-abcd987 |
Obtain a Token
Now that we have a URL, Organisation Id and Cluster Id we finally need a Token before we can install Databricks Connect and then configure it. This is achieved by navigating to your account in Databricks Workspace
Select the option to create a New Token, make sure this is noted as you will need it later
You will be prompted to record the token for future use. The token will be similar to
89E91D0748A846A08853E961E2242832
Install Databricks Connect
We now have the basic details to install and configure Databricks Connect. From the command line run the following:
pip install -U databricks-connect==6.5.*
Configure Databricks Connect
Once Databricks Connect is installed, run the following from the command line
databricks-connect configure
You will now be prompted to enter the following details we have already captured:
Do you accept the above agreement? [y/N] y
Set new config values (leave input empty to accept default):
Databricks Host [no current value, must start with https://]: <databricks-url>
Databricks Token [no current value]: <databricks-token>
Cluster ID (e.g., 0921-001415-jelly628) [no current value]: <cluster-id>
Org ID (Azure-only, see ?o=orgId in URL) [0]: <org-id>
Port [15001]: <port>
The following are the sample values previously collected
| URL | https://adb-1234567891011121314.12.azuredatabricks.net/?o=1234567891011121314#/setting/clusters/0621-093110-abcd987 |
| Token | 89E91D0748A846A08853E961E2242832 |
| Organisation Id | 1234567891011121314 |
| Cluster Id | 0621-093110-abcd987 |
| Port | Simply us the default value of 15001 |
Test Databricks Connect
We have now configured Databricks Connect. From the command line run the following to test:
databricks-connect test
If all proceeds as expected, you will ultimately be presented with an output similar to the following:
All tests passed.
Ok we are now good to go!!
Additional Reading
https://docs.databricks.com/dev-tools/databricks-connect.html
