Rollup Automation¶

Once you get comfortable with manual rollups, you might want to set up regularly executed automated rollups.

At this moment scaraplate doesn’t provide a CLI for that, but there’s a quite extensible Python code which simplifies implementation of custom scenarios.

Supported automation scenarios¶

GitLab Merge Request¶

GitLab integration requires python-gitlab package, which can be installed with:

pip install 'scaraplate[gitlab]'

Sample rollup.py script:

from scaraplate import automatic_rollup
from scaraplate.automation.gitlab import (
    GitLabCloneTemplateVCS,
    GitLabMRProjectVCS,
)


automatic_rollup(
    template_vcs_ctx=GitLabCloneTemplateVCS.clone(
        project_url="https://mygitlab.example.org/myorg/mytemplate",
        private_token="your_access_token",
        clone_ref="master",
    ),
    project_vcs_ctx=GitLabMRProjectVCS.clone(
        gitlab_url="https://mygitlab.example.org",
        full_project_name="myorg/mytargetproject",
        private_token="your_access_token",
        changes_branch="scheduled-template-update",
        clone_ref="master",
    ),
)

This script would do the following:

git clone the template repo to a tempdir;
git clone the project repo to a tempdir;
Run scaraplate rollup ... --no-input;
Would do nothing if rollup didn’t change anything; otherwise it would create a commit with the changes, push it to the scheduled-template-update branch and open a GitLab Merge Request from this branch.

If a MR already exists, GitLabMRProjectVCS does the following:

A one-commit git diff is compared between the already existing MR’s branch and the locally committed branch (in a tempdir). If diffs are equal, nothing is done.
If diffs are different, the existing MR’s branch is removed from the remote, effectively closing the old MR, and a new branch is pushed, which is followed by creation of a new MR.

To have this script run daily, crontab can be used. Assuming that the script is located at /opt/rollup.py and the desired time for execution is 9:00, it might look like this:

$ crontab -e
# Add the following line:
00 9 * * *  python3 /opt/rollup.py

Git push¶

GitLabCloneTemplateVCS and GitLabMRProjectVCS are based off GitCloneTemplateVCS and GitCloneProjectVCS correspondingly. GitLab classes add GitLab-specific git-clone URL generation and Merge Request creation. The rest (git clone, commit, push) is done in the GitCloneTemplateVCS and GitCloneProjectVCS classes.

GitCloneTemplateVCS and GitCloneProjectVCS classes work with any git remote. If you’re okay with just pushing a branch with updates (without opening a Merge Request/Pull Request), then you can use the following:

Sample rollup.py script:

from scaraplate import automatic_rollup, GitCloneProjectVCS, GitCloneTemplateVCS


automatic_rollup(
    template_vcs_ctx=GitCloneTemplateVCS.clone(
        clone_url="https://github.com/rambler-digital-solutions/scaraplate-example-template.git",
        clone_ref="master",
    ),
    project_vcs_ctx=GitCloneProjectVCS.clone(
        clone_url="https://mygit.example.org/myrepo.git",
        clone_ref="master",
        changes_branch="scheduled-template-update",
        commit_author="scaraplate <yourorg@yourcompany>",
    ),
)

Python API¶

scaraplate.automation.base.automatic_rollup(*, template_vcs_ctx: AbstractContextManager[TemplateVCS], project_vcs_ctx: AbstractContextManager[ProjectVCS], extra_context: Optional[Mapping[str, str]] = None) → None¶

The main function of the automated rollup implementation.

This function accepts two context managers, which should return two classes: TemplateVCS and ProjectVCS, which represent the cloned template and target project correspondingly.

The context managers should prepare the repos, e.g. they should create a temporary directory, clone a repo there, and produce a TemplateVCS or ProjectVCS class instance.

This function then applies scaraplate rollup of the template to the target project in no-input mode. If the target project contains any changes (as reported by ProjectVCS.is_dirty()), they will be committed by calling ProjectVCS.commit_changes().

New in version 0.2.

class scaraplate.automation.base.TemplateVCS¶

Bases: abc.ABC

A base class representing a template retrieved from a VCS (probably residing in a temporary directory).

The resulting directory with template must be within a git repository, see Scaraplate Template for details. But it doesn’t mean that it must be retrieved from git. Template might be retrieved from anywhere, it just has to be in git at the end. That git repo will be used to fill the TemplateMeta structure.

dest_path¶: Path to the root directory of the template.

template_meta¶: TemplateMeta filled using the template’s git repo.

class scaraplate.automation.base.ProjectVCS¶

Bases: abc.ABC

A base class representing a project retrieved from a VCS (probably residing in a temporary directory).

The project might use any VCS, at this point there’re no assumptions made by scaraplate about the VCS.

commit_changes(template_meta: scaraplate.template.TemplateMeta) → None¶

Commit the changes made to the project. This method is responsible for delivering the changes back to the place the project was retrieved from. For example, if the project is using git and it was cloned to a temporary directory, then this method should commit the changes and push them back to git remote.

This method will be called only if ProjectVCS.is_dirty() has returned True.

dest_path¶: Path to the root directory of the project.

is_dirty() → bool¶: Tell whether the project has any changes not committed to the VCS.

class scaraplate.automation.git.GitCloneTemplateVCS(template_path: pathlib.Path, template_meta: scaraplate.template.TemplateMeta)¶

Bases: scaraplate.automation.base.TemplateVCS

A ready to use TemplateVCS implementation which:

Uses git
Clones a git repo with the template to a temporary directory (which is cleaned up afterwards)
Allows to specify an inner dir inside the git repo as the template root (which is useful for monorepos)

classmethod clone(clone_url: str, *, clone_ref: Optional[str] = None, monorepo_inner_path: Optional[pathlib.Path] = None) → Iterator[scaraplate.automation.git.GitCloneTemplateVCS]¶

Provides an instance of this class by issuing git clone to a tempdir when entering the context manager. Returns a context manager object which after __enter__ returns an instance of this class.

Parameters:	clone_url – Any valid `git clone` url. clone_ref – Git ref to checkout after clone (i.e. branch or tag name). monorepo_inner_path – Path to the root dir of template relative to the root of the repo. If `None`, the root of the repo will be used as the root of template.

class scaraplate.automation.git.GitCloneProjectVCS(project_path: pathlib.Path, git: scaraplate.automation.git.Git, *, changes_branch: str, commit_author: str, commit_message_template: str)¶

Bases: scaraplate.automation.base.ProjectVCS

A ready to use ProjectVCS implementation which:

Uses git
Clones a git repo with the project to a temporary directory (which is cleaned up afterwards)
Allows to specify an inner dir inside the git repo as the project root (which is useful for monorepos)
Implements ProjectVCS.commit_changes() as git commit + git push.

classmethod clone(clone_url: str, *, clone_ref: Optional[str] = None, monorepo_inner_path: Optional[pathlib.Path] = None, changes_branch: str, commit_author: str, commit_message_template: str = 'Scheduled template update ({update_time:%Y-%m-%d})\n\n* scaraplate version: {scaraplate_version}\n* template commit: {template_meta.commit_url}\n* template ref: {template_meta.head_ref}\n') → Iterator[scaraplate.automation.git.GitCloneProjectVCS]¶

Provides an instance of this class by issuing git clone to a tempdir when entering the context manager. Returns a context manager object which after __enter__ returns an instance of this class.

Parameters:

clone_url – Any valid git clone url.
clone_ref – Git ref to checkout after clone (i.e. branch or tag name).
monorepo_inner_path – Path to the root dir of project relative to the root of the repo. If None, the root of the repo will be used as the root of project.
changes_branch – The branch name where the changes should be pushed in the remote. Might be the same as clone_ref. Note that this branch is never force-pushed. If upon push the branch already exists in remote and its one-commit diff is different from the one-commit diff of the just created local branch, then the remote branch will be deleted and the local branch will be pushed to replace the previous one.
commit_author – Author name to use for git commit, e.g. John Doe <john@example.org>.
commit_message_template –
str.format() template which is used to produce a commit message when committing the changes. Available format variables are:
- update_time [datetime.datetime] – the time of update
- scaraplate_version [str] – scaraplate package version
- template_meta [TemplateMeta] – template meta returned by TemplateVCS.template_meta()

class scaraplate.automation.gitlab.GitLabCloneTemplateVCS(git_clone: scaraplate.automation.git.GitCloneTemplateVCS)¶

Bases: scaraplate.automation.base.TemplateVCS

A class which extends GitCloneTemplateVCS with GitLab-specific clone_url generation.

classmethod clone(project_url: str, private_token: Optional[str] = None, *, clone_ref: Optional[str] = None, monorepo_inner_path: Optional[pathlib.Path] = None) → Iterator[scaraplate.automation.gitlab.GitLabCloneTemplateVCS]¶

Same as GitCloneTemplateVCS.clone() except that clone_url is replaced with project_url and private_token.

The private_token allows to clone private repos, which are visible only for an authenticated user.

Parameters:	project_url – A URL to a GitLab project, e.g. `https://gitlab.example.org/myorganization/myproject`. private_token – GitLab access token, see https://docs.gitlab.com/ce/api/#oauth2-tokens.

class scaraplate.automation.gitlab.GitLabMRProjectVCS(git_clone: scaraplate.automation.git.GitCloneProjectVCS, *, gitlab_project, mr_title_template: str, mr_description_markdown_template: str)¶

Bases: scaraplate.automation.base.ProjectVCS

A class which extends GitCloneProjectVCS with GitLab-specific clone_url generation and opens a GitLab Merge Request after git push.

classmethod clone(gitlab_url: str, full_project_name: str, private_token: str, *, mr_title_template: str = 'Scheduled template update ({update_time:%Y-%m-%d})', mr_description_markdown_template: str = '* scaraplate version: `{scaraplate_version}`\n* template commit: {template_meta.commit_url}\n* template ref: {template_meta.head_ref}\n', commit_author: Optional[str] = None, **kwargs) → Iterator[scaraplate.automation.gitlab.GitLabMRProjectVCS]¶

Same as GitCloneProjectVCS.clone() with the following exceptions:

clone_url is replaced with gitlab_url, full_project_name and private_token.
A GitLab Merge Request (MR) is opened after a successful git push.

The private_token allows to clone private repos, which are visible only for an authenticated user.

As in GitCloneProjectVCS.clone(), the changes_branch might be the same as clone_ref. In this case no MR will be opened.

A MR will be created only if there’re any changes produced by scaraplate rollup. If a changes_branch is already present in remote (i.e. there is a previous automatic rollup which wasn’t merged yet), there’re two possibilities:

If one-commit diffs between the remote’s changes_branch and the local changes_branch are the same, nothing is done. It means that a MR already exists and it has the same patch as the one which was just produced locally.
If the diffs are different, the remote branch will be deleted, effectively closing the old MR, and a new one will be pushed instead, and a new MR will be opened.

The opened MRs are expected to be merged manually.

Parameters:

gitlab_url – A URL to the GitLab instance, e.g. https://gitlab.example.org.
full_project_name – Project name within gitlab, e.g. myorganization/myproject.
private_token –
GitLab access token, see https://docs.gitlab.com/ce/api/#oauth2-tokens.
mr_title_template –
str.format() template which is used to produce a MR title. Available format variables are:
- update_time [datetime.datetime] – the time of update
- template_meta [TemplateMeta] – template meta returned by TemplateVCS.template_meta()
mr_description_markdown_template –
str.format() template which is used to produce a MR description (which will be rendered as markdown). Available format variables are:
- update_time [datetime.datetime] – the time of update
- scaraplate_version [str] – scaraplate package version
- template_meta [TemplateMeta] – template meta returned by TemplateVCS.template_meta()
commit_author – Author name to use for git commit, e.g. John Doe <john@example.org>. If None, will be retrieved from GitLab as the name of the currently authenticated user (using private_token).