CLI Reference

The Michelangelo AI CLI (ma) provides a unified way to manage resources using standard Kubernetes-style commands. This guide covers all supported commands for managing Michelangelo AI API entities.

Command summary

All resource types support get, apply, and delete (see supported resource types below). Additional commands:

Command	Description
ma pipeline run	Execute a registered pipeline
ma pipeline dev-run	Run a pipeline without registering it
ma pipeline delete	Delete a pipeline (cascades to child runs by default)
ma pipeline_run kill	Terminate a running pipeline run
ma trigger_run kill	Terminate a running trigger
ma sandbox create	Set up a local development environment
ma sandbox delete	Tear down the local environment

Supported resource types

Resource Type	CLI Name	Description	Supported Operations
Project	project	Namespace and team ownership for ML resources	get, apply, delete
Pipeline	pipeline	Registered workflow with configuration and scheduling	get, apply, delete, run, dev-run
PipelineRun	pipeline_run	Single execution instance of a pipeline	get, apply, delete, kill
TriggerRun	trigger_run	Scheduled or on-demand pipeline execution trigger	get, apply, delete, kill
Model	model	Trained model artifact with versioning	get, apply, delete
ModelFamily	model_family	Group of related model versions	get, apply, delete
Deployment	deployment	Model serving deployment configuration	get, apply, delete
InferenceServer	inference_server	Runtime server for model inference	get, apply, delete
Revision	revision	Versioned snapshot of a resource	get, apply, delete
Cluster	cluster	Kubernetes cluster configuration	get, apply, delete
RayCluster	ray_cluster	Ray distributed compute cluster	get, apply, delete
RayJob	ray_job	Job submitted to a Ray cluster	get, apply, delete
SparkJob	spark_job	Job submitted to a Spark cluster	get, apply, delete
CachedOutput	cached_output	Cached task output for pipeline resume	get, apply, delete

Note: In Michelangelo AI, a project is the workspace where your pipelines, models, and triggers live. In YAML files and CLI flags, your project is identified by the namespace field — these refer to the same thing. See the Project Management guide for details.

Prerequisites

1. Install Python >= 3.9 and Poetry.

2. Install dependencies:

   cd python/
   poetry install

3. Start the sandbox environment. Follow the Sandbox Setup Guide to install the required software (Docker, kubectl, k3d) and create a local development environment:

   ma sandbox create

   This starts all required services, including the API server (localhost:15566), database, workflow engine, and object storage. See Sandbox commands for the full command reference.

4. (Optional) Configure a custom API server address:

   export MACTL_ADDRESS="127.0.0.1:15566"

   The default address (127.0.0.1:15566) works automatically with the sandbox. Only set this if you are connecting to a different API server instance.

Usage

All Michelangelo AI API entities support the following standard operations -- GET, APPLY, and DELETE

General syntax

cd $REPO_ROOT/python/
ma <RESOURCE_TYPE> <COMMAND> [ARGS]

We will abstract this part like ma <RESOURCE_TYPE> <COMMAND> in below.

GET - Retrieve resource

Retrieve information about an existing resource by project and name. If you don't specify the --name field, it lists all resources under the specified project.

Syntax:

ma <RESOURCE_TYPE> get --namespace="<namespace>" [--name="<name>"]
# Short form: -n for --namespace
ma <RESOURCE_TYPE> get -n "<namespace>" [--name="<name>"]

Examples:

# List all projects
ma project get --namespace="my-project"

# List all pipelines in a project
ma pipeline get --namespace="my-project"

# Get a specific pipeline
ma pipeline get --namespace="my-project" --name="bert-cola-test"

# Get a specific project
ma project get --namespace="my-project" --name="my-project"

# Get a pipeline run
ma pipeline_run get --namespace="my-project" --name="run-001"

Arguments

The following argument is available for list operations (get command without --name):

• --limit [n] - maximum number of results to return (default: 100)

APPLY - Create or update a resource from YAML

Apply (create or update) a resource from a YAML configuration file. The apply command works as an upsert: it creates the resource if it doesn't exist, or updates it if it does. The resource type is automatically detected from the apiVersion and kind fields in the YAML.

Syntax:

ma <RESOURCE_TYPE> apply --file="<YAML_FILE_PATH>"
# Short form: -f for --file
ma <RESOURCE_TYPE> apply -f "<YAML_FILE_PATH>"

Examples:

# Apply a pipeline configuration
ma pipeline apply --file="./examples/bert_cola/pipeline.yaml"

# Apply a project configuration
ma project apply --file="./project.yaml"

DELETE - Remove a resource

Delete a specific resource by project and name.

Syntax:

ma <RESOURCE_TYPE> delete --namespace="<namespace>" --name="<name>"
# Short form: -n for --namespace
ma <RESOURCE_TYPE> delete -n "<namespace>" --name="<name>"

Examples:

# Delete a pipeline
ma pipeline delete --namespace="my-project" --name="bert-cola-test"

# Delete a project
ma project delete --namespace="my-project" --name="my-project"

# Delete a pipeline run
ma pipeline_run delete --namespace="my-project" --name="run-001"

Pipeline delete and cascade

ma pipeline delete removes a Pipeline and cascades to its child PipelineRuns and TriggerRuns (Kubernetes foreground propagation): in-flight runs are drained, their final state is retained in MySQL, then they are removed before the Pipeline. The command prompts for confirmation; pass --yes to skip it. This is irreversible.

ma pipeline delete --namespace="<namespace>" --name="<name>" [--yes]

• --yes — skip the confirmation prompt

# Prompts for confirmation
ma pipeline delete --namespace="my-project" --name="bert-cola-test"

# Skip confirmation (scripting)
ma pipeline delete --namespace="my-project" --name="bert-cola-test" --yes

To delete a Pipeline but keep its runs, use kubectl delete pipeline <name> -n <namespace> --cascade=orphan. For propagation policy, the RBAC caveat, and monitoring, see the Cascade Delete operator guide.

Type-specific commands

Some resource types support additional commands beyond GET, APPLY, and DELETE.

Pipeline

RUN - Execute a pipeline

The RUN command is specifically available for pipelines to create and execute pipeline runs. To run a pipeline, you need to register your pipeline first using ma pipeline apply -f <pipeline_conf.yaml>.

Syntax:

ma pipeline run --namespace="<namespace>" --name="<pipeline_name>"
# Short form: -n for --namespace
ma pipeline run -n "<namespace>" --name="<pipeline_name>"

Example:

# Run a registered pipeline
ma pipeline run --namespace="my-project" --name="bert-cola-test"

Arguments

• --resume_from - create resumed pipeline run from specified pipeline run (specifying resume_from step is optional)

Resume_From Argument

The RUN command also can have a --resume_from argument that allows a new pipeline run to be resumed from a previous pipeline line run. If a pipeline run step is not specified in the resume_from argument, the resumed pipeline will automatically resume from the last failed step of the previous pipeline.

Syntax:

ma pipeline run --namespace="<namespace>" --name="<pipeline_name>" --resume_from=<pipeline_run_name>:<pipeline_run_step_name>

Example:

ma pipeline run --namespace="my-project" --name="bert-cola-test" --resume_from=run-1759873504-b93b7f612:train

Notification Arguments

You can attach notification rules directly to a pipeline run so you're alerted when it reaches a terminal state. This is useful for one-off runs where you want a quick "ping me when it's done" without editing YAML specs.

• --notify-slack — Slack destination (channel or @user). Repeatable or comma-separated.
• --notify-email — Email address. Repeatable or comma-separated.
• --notify-on — Event type to trigger on: SUCCEEDED, FAILED, KILLED, SKIPPED, or STARTED. Repeatable or comma-separated. Defaults to the four terminal states (SUCCEEDED, FAILED, KILLED, SKIPPED) when omitted; STARTED is opt-in. Applies to all destinations (per-destination filtering is not yet supported — use YAML specs for that).

Syntax:

ma pipeline run -n "<namespace>" --name="<pipeline_name>" \
  --notify-slack "<channel_or_user>" \
  --notify-email "<email_address>" \
  --notify-on <EVENT_TYPE>

Example:

# Notify a Slack channel and two email addresses on failure or success
ma pipeline run -n "my-project" --name="bert-cola-test" \
  --notify-slack "#ml-alerts" \
  --notify-email alice@example.com,oncall@example.com \
  --notify-on FAILED,SUCCEEDED

For advanced notification configuration (per-destination event filtering, trigger run notifications, or standing notification rules), see Pipeline Notifications.

DEV RUN - Execute a pipeline in DEV mode

The DEV RUN command is used to run a pipeline without registering it. This command is to allow users to quickly iterate on their pipelines. The dev-run command supports an --env flag for passing environment variables, which are injected into the pipeline's execution environment.

Syntax:

ma pipeline dev-run --file=<YAML_FILE_PATH> --env=<ENV_VAR>=<ENV_VAL>
# Short form: -f for --file
ma pipeline dev-run -f <YAML_FILE_PATH> --env=<ENV_VAR>=<ENV_VAL>

Arguments

• --file / -f - path to the pipeline YAML configuration file (required)
• --env - environment variable to inject (repeatable for multiple variables)
• --file-sync - sync uncommitted local file changes to the remote container
• --storage-url - custom storage URL for file-sync tarballs (e.g., s3://bucket/path)
• --resume_from - resume from a previous pipeline run, optionally specifying a step (<run_name>:<step_name>)

Example:

# Run a pipeline in dev mode
ma pipeline dev-run -f "./examples/bert_cola/pipeline.yaml" --env=foo=bar

# To pass in multiple environment variables:
ma pipeline dev-run -f "./examples/bert_cola/pipeline.yaml" --env=foo=bar --env=lorem=ipsum --env=key=val

Dev-run command with local file sync

Adding --file-sync to the dev-run command enables testing of uncommitted code changes without needing to commit or rebuild Docker images.

# Run a pipeline in dev mode with file sync
ma pipeline dev-run -f "./examples/bert_cola/pipeline.yaml" --env=foo=bar --file-sync

# With custom storage URL
ma pipeline dev-run -f "./examples/bert_cola/pipeline.yaml" --file-sync --storage-url=s3://my-bucket/workflows

Differences between dev-run and remote-run

For an architectural overview of how these two modes differ, see Remote Run vs Pipeline Dev Run: what's actually different.

1. dev-run: Test Pipeline from Local File

pipeline dev-run command runs a pipeline directly from your committed git snapshot. Pipeline run will be controlled by Michelangelo AI API server and controller. This command creates a PipelineRun entity but no Pipeline entity, so you will not see the pipeline entity information in MA Studio.

remote-run (invoked via python my_workflow.py remote-run, not an ma command) bypasses the Michelangelo AI API server and submits your workflow directly to Cadence/Temporal. No Michelangelo AI entities are created, and pipeline status is not visible in MA Studio.

2. dev-run --file-sync: Test Pipeline + Uncommitted Changes

Adding --file-sync to the pipeline dev-run command enables testing of uncommitted code changes without needing to commit or rebuild Docker images.

dev-run --file-sync: Creates two tarballs: a workflow tarball (from committed code) and a file-sync tarball (containing only files changed via git diff). When the container starts, sitecustomize.py downloads the file-sync tarball and overlays changed files on top of the base code. The file-sync URL is passed via the UF_FILE_SYNC_TARBALL_URL environment variable.

remote-run: Creates a workflow tarball from committed code and sends it straight to the workflow engine without creating any Michelangelo AI entities.

remote-run --file-sync: Creates two tarballs: a workflow tarball (base64-encoded in the Cadence CLI input) and a file-sync tarball (uploaded to S3). The S3 URL is passed as an environment variable to the container.

Pipeline_run

Kill - Terminate a pipeline run

The KILL command is used to cleanly terminate a running pipeline. It sets the PipelineRun status to "killed" and aborts the pipeline execution in Cadence/Temporal. The command will prompt for confirmation unless the --yes flag is provided.

Syntax:

ma pipeline_run kill --namespace=<NAMESPACE> --name=<NAME> [--yes]

Parameters:

• --namespace: Kubernetes namespace where the pipeline run exists
• --name: Name of the pipeline run to kill
• --yes: (Optional) Skip confirmation prompt and kill immediately

Example:

# Kill a pipeline run with confirmation prompt
ma pipeline_run kill --namespace=my-project --name=pipeline-run-20251118-194500-8cdb1538

# Kill a pipeline run without confirmation prompt
ma pipeline_run kill --namespace=my-project --name=pipeline-run-20251118-194500-8cdb1538 --yes

Trigger_run

Kill - Terminate a running trigger

The KILL command is used to cleanly terminate a running trigger_run resource. This command sets the trigger's kill flag, which triggers proper Cadence workflow termination. The command will prompt for confirmation unless the --yes flag is provided.

Syntax:

ma trigger_run kill --namespace=<NAMESPACE> --name=<NAME> [--yes]

Example:

# Kill a trigger run with confirmation prompt
ma trigger_run kill --namespace=my-project --name=training-pipeline-cron-trigger

# Kill a trigger run without confirmation prompt
ma trigger_run kill --namespace=my-project --name=training-pipeline-cron-trigger --yes

Sandbox commands

The ma sandbox commands manage a local K3d development environment. For prerequisites, setup walkthrough, and detailed options, see the Sandbox Setup Guide.

Command	Description
ma sandbox create	Create a K3d cluster with all Michelangelo AI services
ma sandbox create --workflow temporal	Create with Temporal instead of Cadence
ma sandbox create --exclude ui	Create without specific services
ma sandbox create --create-compute-cluster	Create with a Ray compute cluster
ma sandbox delete	Tear down the cluster and all resources
ma sandbox start	Start a stopped cluster
ma sandbox stop	Stop the cluster (preserves state)
ma sandbox demo pipeline	Create demo pipeline resources
ma sandbox demo inference	Create demo inference server resources

YAML Resource Examples

Pipeline YAML

apiVersion: michelangelo.api/v2
kind: Pipeline
metadata:
  namespace: "my-project"  # Your project name
  name: "my-pipeline"
spec:
  type: "PIPELINE_TYPE_TRAIN"
  manifest:
    filePath: examples.bert_cola.bert_cola

Project YAML

apiVersion: michelangelo.api/v2
kind: Project
metadata:
  name: my-project
  namespace: my-project
spec:
  description: My ML Project
  owner:
    owningTeam: "michelangelo"
    owners: "sample name"
  tier: 4
  gitRepo: https://github.com/uber/michelangelo
  rootDir: python/michelangelo/cli/sandbox/crds

PipelineRun YAML

apiVersion: michelangelo.api/v2
kind: PipelineRun
metadata:
  name: run-training-pipeline
  namespace: my-project
spec:
  pipeline:
    name: training-pipeline
    namespace: my-project

Configuration

The ma CLI uses a layered configuration system. Settings are resolved in the following priority order (highest to lowest):

1. Environment variables (highest priority)
2. TOML config file (~/.ma/config.toml)
3. Default values (lowest priority)

Configuration file

The configuration file is located at ~/.ma/config.toml and uses TOML format.

Example configuration

[ma]
address = "127.0.0.1:15566"
use_tls = false

[minio]
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
endpoint_url = "http://localhost:9091"

[metadata]
rpc-caller = "grpcurl"
rpc-service = "ma-apiserver"
rpc-encoding = "proto"

[plugin]
dirs = []  # Add custom plugin directories here

Configurable fields

API server

API server configuration is placed under the [ma] section.

• address - Address of the API server (default: 127.0.0.1:15566)
• use_tls - Whether the client uses TLS credentials (default: false)

MinIO credentials

MinIO credentials for object storage are placed under the [minio] section.

• access_key_id - MinIO user name (example: minioadmin)
• secret_access_key - MinIO password (example: minioadmin)
• endpoint_url - MinIO endpoint URL (example: http://localhost:9091)

Custom gRPC metadata

Custom gRPC metadata headers are placed under the [metadata] section.

• rpc-caller - Identifies the calling client (example: grpcurl)
• rpc-service - Target service name (example: ma-apiserver)
• rpc-encoding - Protocol encoding format (example: proto)

Custom plugins

The ma CLI supports custom plugins to extend entity-specific commands and behavior. Plugin configuration is placed under the [plugin] section.

Built-in plugins: The CLI includes built-in plugins located at python/michelangelo/cli/mactl/plugins/entity/ that provide core functionality for entities like pipeline, pipeline_run, and trigger_run.

Custom plugin directories: You can add additional plugin directories by specifying them in the configuration file:

[plugin]
dirs = [
    "/path/to/your/custom/plugins",
    "/another/plugin/directory"
]

Plugin directory structure: Each plugin directory should follow this structure:

your-plugin-directory/
└── entity/
    └── {entity_type}/
        └── main.py

For example, to create a custom pipeline plugin:

my-plugins/
└── entity/
    └── pipeline/
        ├── __init__.py
        └── main.py

Required plugin functions: Plugin modules should implement one or both of these functions:

• apply_plugins(crd: CRD, channel: Channel, *args, **kwargs) - Adds custom command signatures to the entity
• apply_plugin_command(crd: CRD, target_command: str, crds: dict[str, CRD], channel: Channel, *args, **kwargs) - Applies logic for specific commands (e.g., apply, create)

Note: Always include *args, **kwargs in your plugin function signatures. This ensures your plugin remains compatible with future mactl versions that may pass additional context. If it's not used, you may use *_, **__ as a convention to indicate unused parameters.

Note: Support for per-module plugin configuration via plugin.modules is coming soon.

Environment variables

The following environment variables override config file settings:

• MACTL_ADDRESS - Override the API server address
• MACTL_USE_TLS - Override the TLS setting (accepts: true, 1, yes, y)
• AWS_ACCESS_KEY_ID - Override MinIO/S3 access key
• AWS_SECRET_ACCESS_KEY - Override MinIO/S3 secret key
• AWS_ENDPOINT_URL - Override MinIO/S3 endpoint URL

Troubleshooting

Common Issues

1. Connection refused: Ensure the API server is running and accessible
2. Resource not found: Verify project name and resource name are correct
3. YAML parsing errors: Check YAML syntax and required fields
4. Permission denied: Ensure proper authentication/authorization setup

Tips and Best Practices

1. YAML files must include apiVersion, kind, and metadata sections
2. Resource names are case-sensitive and use snake_case in commands (e.g., pipeline_run not PipelineRun)
3. Check API server connectivity if commands fail with gRPC connection errors

Debug Mode

Enable debug logging by setting the environment variable:

export LOG_LEVEL=DEBUG

This will provide detailed information about gRPC calls and internal operations.