Commandline interface

The primary way we expect people to interact with hipercow is via the commandline interface. This page contains the automatically generated reference documentation for this interface; longer explanations are available elsewhere in the documentation.

hipercow

Interact with hipercow.

Usage:

hipercow [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--version`	boolean	Show the version and exit.	`False`
`--help`	boolean	Show this message and exit.	`False`

Subcommands

bundle: Interact with bundles.
create: Commands for task creation.
dide: Commands for interacting with the DIDE cluster.
driver: Configure drivers.
environment: Interact with environments.
init: Initialise a new hipercow root.
repl: Launch the interactive REPL.
task: Create and interact with tasks.

hipercow bundle

Interact with bundles.

Usage:

hipercow bundle [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

delete: Delete a bundle.
list: List bundles.
status: Get the status of a bundle.

hipercow bundle delete

Delete a bundle.

Note that this does not delete the tasks in the bundle, just the bundle itself. So you will not be able to use hipercow bundle to manage the ensemble of jobs together, but you will be able to work with the tasks by their individual ids, using hipercow task.

Usage:

hipercow bundle delete [OPTIONS] NAME

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow bundle list

List bundles.

Usage:

hipercow bundle list [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow bundle status

Get the status of a bundle.

This can offer three levels of summary; and we might redesign the output a bit to make this easier to work with, depending on what people actually do with the output.

Please don't try and parse the output directly, but let us know what sort of format you might like it in, as we can easily add something like a JSON format output.

Usage:

hipercow bundle status [OPTIONS] NAME

Options:

Name	Type	Description	Default
`--summary`	choice (`none` \| `group` \| `single`)	Summarise the statuses	`none`
`--help`	boolean	Show this message and exit.	`False`

hipercow create

Commands for task creation.

Usage:

hipercow create [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

bulk: Bulk create tasks by substituting into a template.

hipercow create bulk

Bulk create tasks by substituting into a template.

These created tasks will then belong to a "bundle" with a name (either automatically generated, or of your choosing), that can be managed using hipercow bundle

The command must contain @-prefixed placeholders such as

mycommand --output=path/@{subdir} @action

which includes the placeholders subdir and action.

You can include data to substitute into this template in three ways:

A single --data=filename.csv argument which will read a csv file of inputs (here it must contain columns subdir and action)
Two arguments --data containing:
a comma separated set of values (e.g., --data action=a,b,c)
a range of values (e.g., --data subdir=0:n or --data subdir=0..n); the : form is python-like and does not include the end of the range, while the .. is inclusive and does include n
in both cases we will compute the outer product of all --data arguments and submit all combinations of arguments.

Usage:

hipercow create bulk [OPTIONS] [CMD]...

Options:

Name	Type	Description	Default
`--data`	text	Data to use in the template	None
`--environment`	text	The environment in which to run the task	None
`--queue`	text	The queue to submit the task to	None
`--name`	text	An optional name for the bundle	None
`--preview`	boolean	Show preview of tasks that would be created, but don't create any	`False`
`--help`	boolean	Show this message and exit.	`False`

hipercow dide

Commands for interacting with the DIDE cluster.

Usage:

hipercow dide [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

authenticate: Interact with DIDE authentication.
check: Check everything is good to use hipercow on the DIDE cluster.

hipercow dide authenticate

Interact with DIDE authentication.

The action can be

set: Set your username and password (the default)
check: Check the stored credentials
clear: Clear any stored credentials

Usage:

hipercow dide authenticate [OPTIONS] [ACTION]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow dide check

Check everything is good to use hipercow on the DIDE cluster.

Usage:

hipercow dide check [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow driver

Configure drivers.

Usage:

hipercow driver [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

configure: Add a driver.
list: List configured drivers.
show: Show configuration for a driver.
unconfigure: Unconfigure (remove) a driver.

hipercow driver configure

Add a driver.

This command will initialise a driver. Most likely you will call

hipercow driver configure dide-windows

because dide-windows is the only real driver at the moment. If you provide the --python-version flag you can specify the Python version to use, if you want a version that is different to the version of Python that you are using locally.

Usage:

hipercow driver configure [OPTIONS] NAME

Options:

Name	Type	Description	Default
`--python-version`	text	Python version to use on the cluster	None
`--help`	boolean	Show this message and exit.	`False`

hipercow driver list

List configured drivers.

Usage:

hipercow driver list [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow driver show

Show configuration for a driver.

Usage:

hipercow driver show [OPTIONS] [NAME]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow driver unconfigure

Unconfigure (remove) a driver.

Usage:

hipercow driver unconfigure [OPTIONS] NAME

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow environment

Interact with environments.

Usage:

hipercow environment [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

delete: Delete an environment.
list: List environments.
new: Create a new environment.
provision: Provision an environment.

hipercow environment delete

Delete an environment.

Usage:

hipercow environment delete [OPTIONS]

Options:

Name	Type	Description	Default
`--name`	text	N/A	None
`--help`	boolean	Show this message and exit.	`False`

hipercow environment list

List environments.

Usage:

hipercow environment list [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow environment new

Create a new environment.

Note that this does not actually install anything; you will need to use

hipercow environment provision

to do that, after creation.

Usage:

hipercow environment new [OPTIONS]

Options:

Name	Type	Description	Default
`--name`	text	Name of the environment	`default`
`--engine`	text	Engine to use	`pip`
`--help`	boolean	Show this message and exit.	`False`

hipercow environment provision

Provision an environment.

This will launch a cluster task that installs the packages you have requested. You can pass a command to run here, or use the defaults if your project has a well known (and well behaved) environment description.

Usage:

hipercow environment provision [OPTIONS] [CMD]...

Options:

Name	Type	Description	Default
`--name`	text	Name of the environment to provision	`default`
`--help`	boolean	Show this message and exit.	`False`

hipercow init

Initialise a new hipercow root.

Create a new hipercow root at the path path. This path should be the root directory of your project (e.g., the path containing .git) and we will create a directory hipercow/ within that directory.

Once initialised, you should configure a driver and environment.

The default is to initialise at the current directory, like git would.

Usage:

hipercow init [OPTIONS] [PATH]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow repl

Launch the interactive REPL.

Running this creates an interactive session where you can send a series of commands to hipercow, with nice autocompletion. Please let us know how you find this.

Type :help for help within the REPL. To quit, type :exit or Ctrl-D.

Usage:

hipercow repl [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow task

Create and interact with tasks.

Usage:

hipercow task [OPTIONS] COMMAND [ARGS]...

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

Subcommands

create: Create a task.
last: List the most recently created task.
list: List all tasks.
log: Get a task log.
recent: List recent tasks.
status: Get the status of a task.
wait: Wait for a task to complete.

hipercow task create

Create a task.

Submits a command line task to the cluster (if you have a driver configured).

The command can be any shell command, though for complex ones we expect that quoting might become interesting - let us know how you get on. If your command involves options (beginning with a -) you will need to use -- to separate the commands to hipercow from those to your application. For example

hipercow task create -- cowsay -t hello

which passes the -t argument through to cowsay. We may remove this requirement in a future version.

If you have multiple environments, you can specify the environment to run the task in with --environment. We validate the presence of this environment at task submission.

If you use --wait then we effectively call hipercow task wait on your newly created task. You can use this to simulate a blocking task create-and-run type loop, but be aware you might wait for a very long time if the cluster is busy.

Usage:

hipercow task create [OPTIONS] [CMD]...

Options:

Name	Type	Description	Default
`--environment`	text	The environment in which to run the task	None
`--queue`	text	Queue to submit the task to	None
`--wait`	boolean	Wait for the task to complete	`False`
`--help`	boolean	Show this message and exit.	`False`

hipercow task last

List the most recently created task.

Usage:

hipercow task last [OPTIONS]

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow task list

List all tasks.

This is mostly meant for debugging; the task list is not very interesting and it might take a while to find them all.

Usage:

hipercow task list [OPTIONS]

Options:

Name	Type	Description	Default
`--with-status`	text	N/A	None
`--help`	boolean	Show this message and exit.	`False`

hipercow task log

Get a task log.

If the log does not yet exist, we return nothing.

Usage:

hipercow task log [OPTIONS] TASK_ID

Options:

Name	Type	Description	Default
`--outer`	boolean	Print the outer logs, from the HPC system	`False`
`--help`	boolean	Show this message and exit.	`False`

hipercow task recent

List recent tasks.

Usage:

hipercow task recent [OPTIONS]

Options:

Name	Type	Description	Default
`--limit`	integer	The maximum number of tasks to list	`10`
`--rebuild`	boolean	Rebuild the recent task list	`False`
`--help`	boolean	Show this message and exit.	`False`

hipercow task status

Get the status of a task.

The task_id will be a 32-character hex string. We print a single status as a result, this might be created, submitted, running, success or failure. Additional statuses will be added in future as we expand the tool.

Usage:

hipercow task status [OPTIONS] TASK_ID

Options:

Name	Type	Description	Default
`--help`	boolean	Show this message and exit.	`False`

hipercow task wait

Wait for a task to complete.

Usage:

hipercow task wait [OPTIONS] TASK_ID

Options:

Name	Type	Description	Default
`--poll`	float	Time to wait between checking on task (in seconds)	`1`
`--timeout`	float	Time to wait for task before failing	None
`--show-log` / `--no-show-log`	boolean	Stream logs to the console, if available?	`True`
`--progress` / `--no-progress`	boolean	Show a progress spinner while waiting?	`True`
`--help`	boolean	Show this message and exit.	`False`

Error handling

If hipercow throws an error, by default only the actual error message is thrown and not the whole stacktrace. Over time, we will try and make these error messages rich enough that you can debug things comfortably with what is shown on screen.

$ hipercow environment new --engine other
Error: Only the 'pip' and 'empty' engines are supported
For more information, run with 'HIPERCOW_TRACEBACK=1'

To get more information you can set the HIPERCOW_TRACEBACK environment variable, either globally (not recommended except for developing hipercow) or locally:

$ HIPERCOW_TRACEBACK=1 hipercow environment new --engine other
Error: Only the 'pip' and 'empty' engines are supported
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/cli.py:54 in cli_safe                      │
│                                                                                                  │
│    51 # handling.                                                                                │
│    52 def cli_safe():                                                                            │
│    53 │   try:                                                                                   │
│ ❱  54 │   │   cli()                                                                              │
│    55 │   except Exception as e:                                                                 │
│    56 │   │   _handle_error(e)                                                                   │
│    57                                                                                            │
│                                                                                                  │
│ ╭───────────────────────────── locals ──────────────────────────────╮                            │
│ │ e = Exception("Only the 'pip' and 'empty' engines are supported") │                            │
│ ╰───────────────────────────────────────────────────────────────────╯                            │
│                                                                                                  │
│ /home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-pac │
│ kages/click/core.py:1161 in __call__                                                             │
│                                                                                                  │
│ /home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-pac │
│ kages/click/core.py:1082 in main                                                                 │
│                                                                                                  │
│ /home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-pac │
│ kages/click/core.py:1697 in invoke                                                               │
│                                                                                                  │
│ /home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-pac │
│ kages/click/core.py:1697 in invoke                                                               │
│                                                                                                  │
│ /home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-pac │
│ kages/click/core.py:1443 in invoke                                                               │
│                                                                                                  │
│ /home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-pac │
│ kages/click/core.py:788 in invoke                                                                │
│                                                                                                  │
│ /home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/cli.py:409 in cli_environment_new          │
│                                                                                                  │
│   406 │                                                                                          │
│   407 │   """                                                                                    │
│   408 │   r = root.open_root()                                                                   │
│ ❱ 409 │   environment_new(r, name, engine)                                                       │
│   410                                                                                            │
│   411                                                                                            │
│   412 @environment.command(                                                                      │
│                                                                                                  │
│ ╭──────────────────────── locals ────────────────────────╮                                       │
│ │ engine = 'other'                                       │                                       │
│ │   name = 'default'                                     │                                       │
│ │      r = <hipercow.root.Root object at 0x7ff0d8893770> │                                       │
│ ╰────────────────────────────────────────────────────────╯                                       │
│                                                                                                  │
│ /home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/environment.py:45 in environment_new       │
│                                                                                                  │
│    42 │                                                                                          │
│    43 │   if engine not in {"pip", "empty"}:                                                     │
│    44 │   │   msg = "Only the 'pip' and 'empty' engines are supported"                           │
│ ❱  45 │   │   raise Exception(msg)                                                               │
│    46 │                                                                                          │
│    47 │   print(f"Creating environment '{name}' using '{engine}'")                               │
│    48 │   EnvironmentConfiguration(engine).write(root, name)                                     │
│                                                                                                  │
│ ╭────────────────────────────────────────── locals ──────────────────────────────────────────╮   │
│ │ engine = 'other'                                                                           │   │
│ │    msg = "Only the 'pip' and 'empty' engines are supported"                                │   │
│ │   name = 'default'                                                                         │   │
│ │   path = PosixPath('/home/rfitzjoh/Documents/src/hipercow-py/tmp/hipercow/py/env/default') │   │
│ │   root = <hipercow.root.Root object at 0x7ff0d8893770>                                     │   │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────╯   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
Exception: Only the 'pip' and 'empty' engines are supported

This uses rich's print_exception functionality to print somewhat human readable stack-traces and information on local variables.

Alternatively you can show the raw Python stack trace with HIPERCOW_RAW_ERROR:

$ HIPERCOW_RAW_ERROR=1 hipercow environment new --engine other
Traceback (most recent call last):
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/bin/hipercow", line 8, in <module>
    sys.exit(cli_safe())
             ~~~~~~~~^^
  File "/home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/cli.py", line 56, in cli_safe
    _handle_error(e)
    ~~~~~~~~~~~~~^^^
  File "/home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/cli.py", line 61, in _handle_error
    raise e
  File "/home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/cli.py", line 54, in cli_safe
    cli()
    ~~~^^
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/rfitzjoh/.local/share/hatch/env/virtual/hipercow/qoF7lZeb/hipercow/lib/python3.13/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/cli.py", line 409, in cli_environment_new
    environment_new(r, name, engine)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/rfitzjoh/Documents/src/hipercow-py/src/hipercow/environment.py", line 45, in environment_new
    raise Exception(msg)
Exception: Only the 'pip' and 'empty' engines are supported

We may ask you to do one of these if reporting an issue.