spark
Manage Spark compute (settings, custom pools)
Authentication scope: fabric
get-settings
Section titled “get-settings”Get workspace-level Spark settings (custom pools, starter pools, etc.)
fabio spark get-settings --workspace <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
Examples
fabio spark get-settings --workspace $WSRead-only · Returns object
update-settings
Section titled “update-settings”Update workspace-level Spark settings
fabio spark update-settings --workspace <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--settings |
string |
No | Settings as JSON (merges with existing settings; conflicts with –runtime-version). Example: ‘{“automaticLog”:{“enabled”:true},“highConcurrency”:{“notebookInteractiveRunEnabled”:true}}’ |
--runtime-version |
string |
No | Set the workspace default Spark runtime version (e.g. “1.3” for Spark 3.5, “2.0” for the Spark 4.1 / Delta 4.2 preview). Merged into existing settings |
Mutates state · Returns object
list-pools
Section titled “list-pools”List custom Spark pools in a workspace
fabio spark list-pools --workspace <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
Read-only · Returns list
get-pool
Section titled “get-pool”Show details of a custom Spark pool
fabio spark get-pool --workspace <value> --pool-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--pool-id |
string |
Yes | Pool ID |
Read-only · Returns object
create-pool
Section titled “create-pool”Create a custom Spark pool
fabio spark create-pool --workspace <value> --name <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--name |
string |
Yes | Pool name |
--node-family |
string |
No | Node family (e.g., MemoryOptimized) |
--node-size |
enum |
No | Node size (e.g., Small, Medium, Large, XLarge, XXLarge) One of: Small, Medium, Large, XLarge, XXLarge. |
--auto-scale-enabled |
bool |
No | Enable auto-scale |
--min-node-count |
integer |
No | Minimum node count (when auto-scale is enabled) |
--max-node-count |
integer |
No | Maximum node count |
--dynamic-executor-enabled |
bool |
No | Enable dynamic executor allocation |
--min-executors |
integer |
No | Minimum executors for dynamic allocation |
--max-executors |
integer |
No | Maximum executors for dynamic allocation |
Examples
fabio spark create-pool --workspace $WS --content '{"name":"HighMem","nodeFamily":"MemoryOptimized","nodeSize":"Large","autoScale":{"enabled":true,"minNodeCount":1,"maxNodeCount":10}}'Mutates state · Returns object
update-pool
Section titled “update-pool”Update a custom Spark pool
fabio spark update-pool --workspace <value> --pool-id <value> --config <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--pool-id |
string |
Yes | Pool ID |
--config |
string |
Yes | Pool configuration as JSON (partial update) |
Mutates state · Returns object
delete-pool
Section titled “delete-pool”Delete a custom Spark pool
fabio spark delete-pool --workspace <value> --pool-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--pool-id |
string |
Yes | Pool ID |
Mutates state · Destructive · Returns void
get-capacity-settings
Section titled “get-capacity-settings”Get capacity-level Spark settings
fabio spark get-capacity-settings --capacity-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
Read-only · Returns object
update-capacity-settings
Section titled “update-capacity-settings”Update capacity-level Spark settings
fabio spark update-capacity-settings --capacity-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
--file |
string |
No | JSON file path with settings |
--content |
string |
No | JSON content with settings (inline) |
Mutates state · Returns void
list-capacity-pools
Section titled “list-capacity-pools”List custom Spark pools in a capacity
fabio spark list-capacity-pools --capacity-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
Read-only · Returns list
create-capacity-pool
Section titled “create-capacity-pool”Create a custom Spark pool in a capacity
fabio spark create-capacity-pool --capacity-id <value> --name <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
--name |
string |
Yes | Pool name |
--file |
string |
No | JSON file path with pool configuration |
--content |
string |
No | JSON content with pool configuration (inline) |
Mutates state · Returns object
get-capacity-pool
Section titled “get-capacity-pool”Get details of a capacity Spark pool
fabio spark get-capacity-pool --capacity-id <value> --pool-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
--pool-id |
string |
Yes | Pool ID |
Read-only · Returns object
update-capacity-pool
Section titled “update-capacity-pool”Update a capacity Spark pool
fabio spark update-capacity-pool --capacity-id <value> --pool-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
--pool-id |
string |
Yes | Pool ID |
--file |
string |
No | JSON file path with pool configuration |
--content |
string |
No | JSON content with pool configuration (inline) |
Mutates state · Returns void
delete-capacity-pool
Section titled “delete-capacity-pool”Delete a capacity Spark pool
fabio spark delete-capacity-pool --capacity-id <value> --pool-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--capacity-id |
string |
Yes | Capacity ID |
--pool-id |
string |
Yes | Pool ID |
Mutates state · Destructive · Returns void
list-livy-sessions
Section titled “list-livy-sessions”List Livy sessions in a workspace
fabio spark list-livy-sessions --workspace <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
Read-only · Returns list
get-livy-session
Section titled “get-livy-session”Get details of a Livy session
fabio spark get-livy-session --workspace <value> --livy-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--livy-id |
string |
Yes | Livy session ID |
Read-only · Returns object
create-livy-session
Section titled “create-livy-session”Create an interactive Livy session on a lakehouse to run Spark code
fabio spark create-livy-session --workspace <value> --lakehouse <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID the session attaches to |
--name |
string |
No | Optional session display name |
--conf |
string |
No | Spark configuration as a JSON object, e.g. ‘{“spark.executor.cores”:“4”}’ |
--wait |
bool |
No | Wait until the session reaches the idle (ready) state before returning |
--timeout |
integer |
No | Max seconds to wait for idle when –wait is set (default 300) |
Mutates state · Returns object
run-statement
Section titled “run-statement”Run a statement (Spark/PySpark/Spark-SQL/R) on an existing Livy session
fabio spark run-statement --workspace <value> --lakehouse <value> --session-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID the session belongs to |
--session-id |
string |
Yes | Interactive Livy session ID (from create-livy-session) |
--code |
string |
No | Code to run (inline, @file, or piped via stdin) |
--language |
enum |
No | Language of the code One of: pyspark, scala, sql, r. |
--timeout |
integer |
No | Max seconds to wait for the statement to finish (default 300) |
Mutates state · Returns object
submit-statement
Section titled “submit-statement”Submit a statement WITHOUT waiting (async) — returns the statement id to poll/cancel
fabio spark submit-statement --workspace <value> --lakehouse <value> --session-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID the session belongs to |
--session-id |
string |
Yes | Interactive Livy session ID |
--code |
string |
No | Code to run (inline, @file, or piped via stdin) |
--language |
enum |
No | Language of the code One of: pyspark, scala, sql, r. |
Mutates state · Returns object
get-statement
Section titled “get-statement”Get the current state and output of a statement (read-only; poll until available)
fabio spark get-statement --workspace <value> --lakehouse <value> --session-id <value> --statement-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID the session belongs to |
--session-id |
string |
Yes | Interactive Livy session ID |
--statement-id |
string |
Yes | Statement ID (from submit-statement/run-statement) |
Read-only · Returns object
cancel-statement
Section titled “cancel-statement”Cancel a running statement (stops the Spark compute it is consuming)
fabio spark cancel-statement --workspace <value> --lakehouse <value> --session-id <value> --statement-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID the session belongs to |
--session-id |
string |
Yes | Interactive Livy session ID |
--statement-id |
string |
Yes | Statement ID to cancel |
Mutates state · Returns object
delete-livy-session
Section titled “delete-livy-session”Delete (stop) an interactive Livy session
fabio spark delete-livy-session --workspace <value> --lakehouse <value> --session-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID the session belongs to |
--session-id |
string |
Yes | Interactive Livy session ID |
Mutates state · Destructive · Returns void
One-shot: create a session, run code, return output, then delete the session
fabio spark run --workspace <value> --lakehouse <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--lakehouse |
string |
Yes | Lakehouse ID to run on |
--code |
string |
No | Code to run (inline, @file, or piped via stdin) |
--language |
enum |
No | Language of the code One of: pyspark, scala, sql, r. |
--conf |
string |
No | Spark configuration as a JSON object |
--high-concurrency |
bool |
No | Use a High-Concurrency session (shared underlying Spark session). Concurrent runs with the same –session-tag pack onto one Spark session |
--session-tag |
string |
No | Session tag for High-Concurrency packing (implies –high-concurrency semantics; requires –high-concurrency). Runs sharing a tag share one Spark session |
--timeout |
integer |
No | Max seconds to wait for the session to be ready and the statement to finish (default 300) |
Mutates state · Returns object
get-advice
Section titled “get-advice”Get Spark advisor real-time advice for a Spark application (monitoring API)
fabio spark get-advice --workspace <value> --item-type <value> --item-id <value> --livy-id <value> --app-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--item-type |
enum |
Yes | The item type that ran the Spark application One of: notebook, spark-job-definition, lakehouse. |
--item-id |
string |
Yes | The item ID (notebook / Spark job definition / lakehouse) |
--livy-id |
string |
Yes | Livy session ID |
--app-id |
string |
Yes | Spark application ID (e.g. application_1741176604085_0001) |
--attempt-id |
string |
No | Application attempt ID (defaults to the last attempt) |
Read-only · Returns object
get-resource-usage
Section titled “get-resource-usage”Get the resource usage timeline for a Spark application (monitoring API)
fabio spark get-resource-usage --workspace <value> --item-type <value> --item-id <value> --livy-id <value> --app-id <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--item-type |
enum |
Yes | The item type that ran the Spark application One of: notebook, spark-job-definition, lakehouse. |
--item-id |
string |
Yes | The item ID (notebook / Spark job definition / lakehouse) |
--livy-id |
string |
Yes | Livy session ID |
--app-id |
string |
Yes | Spark application ID |
--attempt-id |
string |
No | Application attempt ID (defaults to the last attempt) |
Read-only · Returns object
get-logs
Section titled “get-logs”Get Spark driver / executor / livy logs for a Spark application (monitoring API)
fabio spark get-logs --workspace <value> --item-type <value> --item-id <value> --livy-id <value> --type <value>| Flag | Type | Required | Description |
|---|---|---|---|
--workspace |
string |
Yes | Workspace ID |
--item-type |
enum |
Yes | The item type that ran the Spark application One of: notebook, spark-job-definition, lakehouse. |
--item-id |
string |
Yes | The item ID (notebook / Spark job definition / lakehouse) |
--livy-id |
string |
Yes | Livy session ID |
--type |
enum |
Yes | Log type One of: driver, executor, livy. |
--app-id |
string |
No | Spark application ID (not required for –type livy) |
--attempt-id |
string |
No | Application attempt ID (defaults to the last attempt) |
--file-name |
string |
No | Specific log file name (e.g. stderr, stdout) — required for driver/executor content |
--meta |
bool |
No | Return log file metadata (JSON) instead of the raw log content |
Read-only · Returns object