Databricks classic compute resources run in the customer’s subscription, using the new serverless warehouse, it is run in Databrick’s own subscription. A pool of running Kubernetes containers which is managed by Databricks, is assigned to a user within seconds.
Normal starting a new classical cluster takes up to 10 minutes, Databricks can assign a serverless running cluster within seconds. With these new cluster type it won’t be necessary to run clusters 24*7, the auto shutdown can be e.g. configured to 10 minutes, saving a large part of the cost necessary to provide good query performance to all users.
As of this release, the Serverless data plane is used only for serverless SQL warehouses. Enabling this feature does not change how Databricks Runtime clusters work in the Data Science & Engineering or Databricks Machine Learning environments.
