Introduction

Git all remotes (gitar) is a command line tool that brings common development operations such as opening a pull request down to the shell. It works with multiple domains such as Github and Gitlab. It can be seen as an alternative to both Github https://github.com/cli/cli and Gitlab https://gitlab.com/gitlab-org/cli cli tools. The scope is smaller as the main focus is not to implement all the workflows but the ones a software engineer might use more often, such as opening merge requests or quickly gathering information about important processes like the status of a pipeline or releases.

Gitar implements the REST API for Gitlab and Github following best practices, such as respecting rate limits and pagination. It also implements local caching to minimize interacting with the REST API and it uses conditional HTTP requests when the local cache is considered expired.

The tool is written in Rust and it is available for Linux, MacOS.

Sections

Getting Started

Get started by installing the binary and creating a configuration file.

Gitar commands

List of commands available in Gitar.

Getting Started

In order to get started using gitar, you'll need to install the binary and setup a configuration file. Let's get started:

• Installation
• Configuration
• Caching
• Listing data

Installation

• Linux
• MacOS

There are multiple ways to install gitar. The easiest would be to go to the gitar releases page at https://github.com/jordilin/gitar/releases and download the tarball for your platform.

Linux

Download the tar.gz file that is built using musl libc. This should work on all linux distributions as it provides a static binary with no platform dependencies.

For example, in the last release v0.1.62 we could download https://github.com/jordilin/gitar/releases/download/v0.1.62/gr-x86_64-unknown-linux-musl.tar.gz, that would correspond for a Linux x86_64 machine.

MacOS

Similarly, for MacOS, download the tar.gz file that corresponds to your CPU architecture and is built for Darwin. The last release v0.1.62 for a Mac computer with the M processor would be https://github.com/jordilin/gitar/releases/download/v0.1.62/gr-aarch64-apple-darwin.tar.gz and for an Intel one would be https://github.com/jordilin/gitar/releases/download/v0.1.62/gr-x86_64-apple-darwin.tar.gz

Configuration

• A quick note about API calls and pages
• Create a configuration file (Optional)
• TOML domain sections
  • No configuration file
• API token
  • Environment variable
  • Github.com
  • Gitlab.com
• Merge requests configuration section
  • Per project merge request configurations
• API types and their configurations
  • Maximum pages to retrieve per API type
  • Local cache duration for each API type
• Split configuration files

A quick note about API calls and pages

When I talk about an API call or number of pages, I'm referring to actual HTTP requests to the Github or Gitlab API. Hence, when I say an API call or one page of information, I'm referring to a single HTTP request.

Create a configuration file (Optional)

In order to create a configuration file, you can run the following command. In this example we are setting the domain to github.com. It can be gitlab.com or any other domain that you want to use, for example your own company's domain.

gr init --domain github.com

This will create a configuration file in your $HOME/.config/gitar/gitar.toml directory with some defaults. The configuration follows a TOML file format. Once the file is created, open the configuration file to add your API token and the optional sections.

TOML domain sections

The configuration file is divided into sections. Each section is named after the domain you are targetting. The formatting of the sections is as follows. Dots in the domain name are replaced by underscores.

[ github_com ]
api_token="<your token>"

[ gitlab_com ]
api_token="<your token>"

[ gitlab_yourcompany_com ]
api_token="<your token>"

No configuration file

Potential use cases: CI/CD pipelines, automation scripts, one-off runs.

If no configuration file is provided, then gitar expects an authentication token environment variable to be set. Check the API token section for more information.

No configuration means that there won't be any caching for read operations either.

API token

Gitar needs an API token to access the Github or Gitlab API. The token can be set by using an environment variable or by placing it in the configuration file.

Environment variable

The environment variable needs to be named after the domain you are targeting. For example, if the remote is gitlab.com the environment variable should be named GITLAB_API_TOKEN and if the remote is github.com the environment variable should be named GITHUB_API_TOKEN. If you have a subdomain such as gitlab.yourcompany.com, the environment variable should be named GITLAB_YOURCOMPANY_API_TOKEN. Finally, if you are using a non FQDN domain such as mygitlab, then the environment variable should be MYGITLAB_API_TOKEN.

This can be summarized in the following table:

If the environment variable is not set, then Gitar will look for the token in the configuration file as explained in the following sections.

Github.com

To get an API token for Github, go to your Github account settings -> Developer settings -> Personal access tokens -> Tokens (classic) At the time of writing, the URL is https://github.com/settings/tokens

Create a new token with the scopes: repo, user, project, gist. By clicking on each scope check box it will automatically select all the sub-scopes under it. Then copy the token and place it in the configuration file. You'll see a line like:

[ github_com ]
api_token="<your-token>"

Gitlab.com

To get an API token for Gitlab, go to your Gitlab account settings -> Access tokens and create an api token. Current URl at the time of writing is https://gitlab.com/-/user_settings/personal_access_tokens Select the api scope, give it a name and an expiration date. Click on Create personal access token and copy the token over to the configuration file.

[ gitlab_com ]
api_token="<your-token>"

Merge requests configuration section

The assignee username is the username that will be used to automatically assign a pull request to. Normally, that would be your username. Example, whenever I create a pull request to my own repository, I automatically assign it to myself. Members are the members you want to potentially assign a merge request to. For Gitlab, both members and assignee need to be formatted with a map of username and user ID. For Github, only the username is needed but the user ID can also be provided. Gitar can pull this information directly from the API:

# Retrieve number of pages required to retrieve candidates for assignee assignment
gr pj members --num-pages
# You might want to bypass throttling if num pages is just a few of them (<10)
gr pj members --from-page 1 --to-page <total-pages> --throttle 2000 --format toml | tee members.toml
# Retrieve my username metadata.
gr us get <my-username> --format toml | tee my-username.toml

The output can be pasted to the configuration file. NOTE: The user ID number can be placed in between quotes or without them.

[ github_com.merge_requests ]
preferred_assignee_username={ "username" = "<your-github-username>", "id" = <your-github-user-id> }
members = [
  { username = "user1", id = "1234" },
  { username = "user2", id = "5678" },
  { username = "user3", id = "9012" }
]

[ gitlab_com.merge_requests ]
preferred_assignee_username={ "username" = "<your-gitlab-username>", "id" = <your-gitlab-user-id> }
members = [
  { username = "user1", id = "1234" },
  { username = "user2", id = "5678" },
  { username = "user3", id = "9012" },
]

Per project merge request configurations

If you want to have different members in different projects, you can do so by adding the following section [<domain>.merge_requests.<group>_<project_name>]. Basically the path / is replaced by _. Ex. jordilin/gitar becomes jordilin_gitar. Same for subgroups: group_subgroup_projectname.

Example:

[ github_com.merge_requests.jordilin_gitar ]
members = []

This will effectively override the global configuration for the domain.

API types and their configurations

Gitar groups API calls into different types taking full control on how we want to retrieve information and how long it is going to be cached. Why is that? The reason is that as project owners or collaborators of the projects we work on, we know in advance the rate of change. Project information such as its members, don't get added or removed on a daily basis, so we can cache that information for a long time. On the other hand, the status of a pipeline, releases, merge requests change more often. The number of pages to retrieve per API can also be adjusted. Please see section caching for more information.

API types:

• Project
• Merge request
• Pipeline
• Release
• Container registry
• Repository tags

Maximum pages to retrieve per API type

One page equals to one HTTP request. Gitar has an internal default of 10 maximum pages that can be retrieved per API call. This takes effect on list operations in every subcommand that has listing support. This can be increased/decreased on a per API basis. This information needs to be set in the TOML section [domain.max_pages_api]. Ex. [github_com.max_pages_api].

• project=<number> This API type is used to retrieve information about a project/repository such as its members. When opening a merge request gitar will pull up to project pages of members to find the your username to assign the pull request to. If you get an error that your username cannot be found, increase this number. Once the members have been retrieved, the list is permanently cached for next calls, so it will be fast.

• merge_request=<number> This API type is used to retrieve information about pull/merge requests. For example, listing opened, merged, closed pull requests, etc...

• pipeline=<number> This API type is used to retrieve information about CI/CD pipelines/actions that run in the given project. This takes place in list operations in the pp subcommand.

• release=<number> This API type is used to retrieve information about releases in the current project, such as listing releases and its assets.

• container_registry=<number> This API type is used to retrieve information about container registry images in the current project. This is supported in Gitlab only. This takes place in list operations in the dk subcommand.

• repository_tags=<number> This API type is used to retrieve information about tags in a repository. This takes place when listing repository tags using the gr pj tags subcommand.

Local cache duration for each API type

Gitar has local caching support for each API type. Every HTTP response is cached and the next time the same request is made, the same response is returned until expired. The responses are stored in a local cache directory which can be configured by setting the cache location in the [<domain>] section

• cache_location="<full-path-to-cache-directory>" The path needs to exist and be writable by the user running the gitar command.

Cache values are a number followed by a letter representing the time unit. For example 5m means 5 minutes, 5d means 5 days, 30s means 30 seconds. The units supported are s for seconds, m for minutes, h for hours, d for days. A cache value of 0 followed by a time unit means automatic expiration of the cache. In that case, gitar will contact the remote API doing a conditional HTTP request to check if the cache is still valid and return the cached response if it is. Otherwise, it will automatically update the cache with the new response.

Cache duration for each API type can be set in the TOML section [<domain>.cache_expirations]. Ex. [github_com.cache_expirations].

• merge_request="<number><time-unit>" This API type is used to retrieve information about pull/merge requests. For example, listing opened, merged, closed pull requests.

• project="<number><time-unit>" This API type is used to retrieve information about a project/repository such as its members. When opening a merge request gitar will pull up to max_pages_api_project pages information to retrieve project information and its members. Project information does not change often, so a higher cache value of a few days can be ok. Members of a project, project ID, etc... can be cached for longer time depending on the projects you work on.

• pipeline="<number><time-unit>" This API type is used to retrieve information about CI/CD pipelines/actions that run in the given project. A low cache value is recommended for this API type as the status of pipelines change often in projects.

• "container_registry="<number><time-unit>" This API type is used to retrieve information about container registry images in the current project. This is supported in Gitlab only. This takes place in list operations in the dk subcommand.

• release="<number><time-unit>" This API type is used to retrieve information about releases in the current project, such as listing releases and its assets.

• single_page="<number><time-unit>" This API type is used to retrieve information about single page calls. For example, trending repositories in github.com. A value of 1d is recommended for this API type.

• repository_tags="<number><time-unit>" This API type is used to retrieve information about tags in a repository.

Note: Local cache can be automatically expired and refreshed by issuing the -r flag when running the gr command.

Split configuration files

If you have merge request configuration for multiple projects, multiple domains, the main configuration file gitar.toml can quickly grow in size. To avoid this, you can split the configuration file into multiple files as follows. Gitar reads the main configuration file gitar.toml and then attempts to read the following file name patterns in the same directory:

• <domain>.toml Ex: github_com.toml, gitlab_com.toml, gitlab_yourcompany_com.toml
• <domain>_<group>_<project>.toml Ex: github_com_jordilin_gitar.toml, gitlab_com_group_subgroup_projectname.toml

As we can observe in the examples above, the following conventions are used:

1. Substitute . with _.
2. Substitute / with _ for domain, group, and project names.

The total configuration is the concatenation of all the files. For example, if we have gitar.toml and github_com.toml in the same directory, then gitar will read both files and concatenate the configuration. If there are duplicate sections it will throw a TOML configuration error. Sections can be added in any of the files. For example, if you were to specify the api_token for Github in github_com.toml adding it to the gitar.toml file would be an error. If you prefer, you can also keep one configuration for each domain and remove the main gitar.toml file.

Gitar's caching configuration approach

Every HTTP API call to Gitlab or Github is categorized into different API operation types. Each operation type has its own cache duration and that is defined by the user in the configuration file. The reason for this is that some resources like the project members in the repositories that you collaborate on might not change often, while pipelines and merge requests change way more often. If you setup the cache to be "0<time_unit>" whichever time unit you want (e.g 0s, 0m, 0h, 0d), then regular HTTP caching mechanisms will take place. In this case, Gitar will cache and then inspect the cache-control header and its directives to determine the cache state and if it should be invalidated or not. While it will perform better than no cache, it won't perform as fast as just immediately returning the cached response as mandated by the user. If you know up front that some resources don't change often, you can set the cache duration to a higher value and then Gitar will return the cached response immediately without making additional HTTP calls.

Use cases:

• Opening merge requests, project information can be cached for a long time making assignee lookups nearly immediate
• Data extraction/experimentation. If you are going to gather release data, merge requests, etc... you can cache the responses for a long time for faster experimentation.

Evaluation order of cache duration

1. Look for the API type specific cache duration (determined by the user)
2. If not found or configured to be "0<time_unit>", then inspect the HTTP cache-control header and its directives to determine the cache state.

All in all, the user is in full control for how long the cache should be kept for while still respecting HTTP cache control mechanisms.

The list subcommand

• Auto throttling
• Max pages to fetch

The list subcommand is used to pull data from specific resources such as pipelines and merge requests. Gitar implements best practices to avoid being rate limited, caches responses and uses pagination to pull the required data using the --from-page and --to-page flags.

Auto throttling

Gitar will automatically throttle the requests after three consecutive HTTP calls have been made. The throttling is based on the rate limit headers plus a jitter interval between 1 and 5 seconds. The user can also specify a fixed throttle interval with --throttle or a random one with --throttle-range.

Max pages to fetch

If no configuration is provided, the default is a max of 10 pages. This can be overriden with --to-page where it will fetch up to the specified page or a range of pages with --from-page and --to-page.

Gitar commands

• Commands available
• Global options
• List options

Commands available

• Merge requests
• Pipelines
• Amps

All gitar commands have a set of common options that can be used to control their behavior.

Global options

• --help - Show help message and exit.
• --version - Show version information and exit.
• --verbose - Enable logging of debug messages. This is useful for debugging issues with the tool. Log traces are written to the standard error output.

List options

List options control the behavior of gitar commands that list resources. They enable throttling, pagination and sorting of the output.

They can be found under the List options section when issuing --help.

A resource such as a pipeline in Gitlab or action in Github can have a large number of items. The list options allows us to retrieve just one page, or a subset of the items by controlling the --from-page and --to-page options.

Useful options when listing resources are:

• --num-pages The total number of pages available to retrieve. If the resource contains lots of items, we can issue gitar with throttling enabled in order to avoid hitting the API rate limit.

gr mr

gr mr is probably one of the most used commands in gitar as it allows you to open and handle merge requests from the command line. It supports several subcommands that allow you to list, create, update, and merge merge requests.

Open a merge request

In its most basic form, you just create a new merge request with the following command:

gr mr create

This assumes you are in a feature branch and you want to merge it into the default branch in origin. The command will prompt you for the title, description, assignee and finally confirm if you want to create a merge request.

gr pp

gr pp is a command that allows you to handle pipelines from the command line.

• List pipelines
• Lint pipeline configuration (.gitlab-ci.yml)
• Get runners available for the project (Gitlab)
• Get the merged .gitlab-ci.yml
• Mermaid diagram of .gitlab-ci.yml

List pipelines

To list pipelines for the current project, you can use the following command:

gr pp list

Lint pipeline configuration (.gitlab-ci.yml)

To lint the pipeline configuration file (.gitlab-ci.yml), you can use the following command:

gr pp lint

Get runners available for the project (Gitlab)

To get the runners available for the project, you can use the following command:

gr pp rn list <status>

Where <status> can be one of the following values:

• online
• offline
• stale
• never-contacted
• all

Get the merged .gitlab-ci.yml

In the scenario where you use a Gitlab pipeline declared in .gitlab-ci.yml and the pipeline contains include statements, you can use the following command to obtain the total configuration of the pipeline:

gr pp merged-ci

This will print out to the console the total merged .gitlab-ci.yml file that includes all the contents from the included yaml files. If the pipeline has errors it will print out the errors if any.

Mermaid diagram of .gitlab-ci.yml

This command is intended to provide a quick and general idea about the project's pipeline structure by computing a state Mermaid diagram off of the .gitlab-ci.yml contents. The implementation is not exhaustive and does not cover all the possible cases that can govern a pipeline creation. It just computes the stages involved, its jobs and the links in between them. The command does not compute whether a stage belongs to a specific branch or to a merge request event.

gr pp chart

For example, given the following .gitlab-ci.yml:

stages:
  - build
  - tests
  - deploy

compile:
  stage: build
  script:
    - echo "build the project"

unittests:
  stage: tests
  script:
    - echo "Run unit tests"

integration:
  stage: tests
  script:
    - echo "Run integration tests"

deploy_job:
  stage: deploy
  script:
    - echo "Deploying the project..."

The corresponding Mermaid diagram that would get generated would look like this:

stateDiagram-v2
    direction LR
    state build{
        direction LR
        state "jobs" as anchorT0
        state "compile" as anchorT0
    }
    build --> tests
    state tests{
        direction LR
        state "jobs" as anchorT1
        state "unittests" as anchorT1
        state "integration" as anchorT1
    }
    tests --> deploy
    state deploy{
        direction LR
        state "jobs" as anchorT2
        state "deploy_job" as anchorT2
    }

If you copy the above diagram and paste it in a markdown file or in the mermaid live editor https://mermaid.live, you will get the following diagram:

gr amps

gr amps lists and execute amps. Amps are wrappers around the gr command itself. Amps normally execute gr subcommands and perform additional logic to get to the desired result. They can also be seen as just gr scripts that can be executed from gr itself. A curated list of amps is provided at https://github.com/jordilin/gitar-amps.

List available amps

gr amps

or

gr amps list

Execute an amp in-line

To execute an amp in-line, you can use the following command:

gr amps exec "<amp-name> <arg_0> <arg_1> ... <arg_n>"

For example:

gr amps exec "list-last-assets github.com/jordilin/gitar"

will print out the URLs of the last stable release assets for the github.com/jordilin/gitar repository.

> Note: Arguments for the amps are optional and the amp name and its arguments should be enclosed in double quotes.

Execute an amp by prompt

gr amps exec

This command will prompt you to select an amp from the list of available amps. After selecting the amp name, it will prompt you to enter the arguments for the amp. Upon pressing enter, the amp will be executed.

The prompt undertands the following prompt queries once an amp has been selected:

• h or help - Show help message for the selected amp.
• q or quit - Quit the prompt and return back to the CLI.