Rollup Automation¶
Once you get comfortable with manual rollups, you might want to set up regularly executed automated rollups.
At this moment scaraplate doesn’t provide a CLI for that, but there’s a quite extensible Python code which simplifies implementation of custom scenarios.
Supported automation scenarios¶
GitLab Merge Request¶
GitLab integration requires python-gitlab package, which can be installed with:
pip install 'scaraplate[gitlab]'
Sample rollup.py
script:
from scaraplate import automatic_rollup
from scaraplate.automation.gitlab import (
GitLabCloneTemplateVCS,
GitLabMRProjectVCS,
)
automatic_rollup(
template_vcs_ctx=GitLabCloneTemplateVCS.clone(
project_url="https://mygitlab.example.org/myorg/mytemplate",
private_token="your_access_token",
clone_ref="master",
),
project_vcs_ctx=GitLabMRProjectVCS.clone(
gitlab_url="https://mygitlab.example.org",
full_project_name="myorg/mytargetproject",
private_token="your_access_token",
changes_branch="scheduled-template-update",
clone_ref="master",
),
)
This script would do the following:
git clone
the template repo to a tempdir;git clone
the project repo to a tempdir;- Run
scaraplate rollup ... --no-input
; - Would do nothing if rollup didn’t change anything; otherwise it would
create a commit with the changes, push it to the
scheduled-template-update
branch and open a GitLab Merge Request from this branch.
If a MR already exists, GitLabMRProjectVCS
does the following:
- A one-commit git diff is compared between the already existing MR’s branch and the locally committed branch (in a tempdir). If diffs are equal, nothing is done.
- If diffs are different, the existing MR’s branch is removed from the remote, effectively closing the old MR, and a new branch is pushed, which is followed by creation of a new MR.
To have this script run daily, crontab can be used. Assuming that the script
is located at /opt/rollup.py
and the desired time for execution is 9:00,
it might look like this:
$ crontab -e
# Add the following line:
00 9 * * * python3 /opt/rollup.py
Git push¶
GitLabCloneTemplateVCS
and
GitLabMRProjectVCS
are based off
GitCloneTemplateVCS
and
GitCloneProjectVCS
correspondingly. GitLab classes add GitLab-specific git-clone URL
generation and Merge Request creation. The rest (git clone, commit, push)
is done in the GitCloneTemplateVCS
and GitCloneProjectVCS
classes.
GitCloneTemplateVCS
and
GitCloneProjectVCS
classes work
with any git remote. If you’re okay with just pushing a branch with
updates (without opening a Merge Request/Pull Request), then you can
use the following:
Sample rollup.py
script:
from scaraplate import automatic_rollup, GitCloneProjectVCS, GitCloneTemplateVCS
automatic_rollup(
template_vcs_ctx=GitCloneTemplateVCS.clone(
clone_url="https://github.com/rambler-digital-solutions/scaraplate-example-template.git",
clone_ref="master",
),
project_vcs_ctx=GitCloneProjectVCS.clone(
clone_url="https://mygit.example.org/myrepo.git",
clone_ref="master",
changes_branch="scheduled-template-update",
commit_author="scaraplate <yourorg@yourcompany>",
),
)
Python API¶
-
scaraplate.automation.base.
automatic_rollup
(*, template_vcs_ctx: AbstractContextManager[TemplateVCS], project_vcs_ctx: AbstractContextManager[ProjectVCS], extra_context: Optional[Mapping[str, str]] = None) → None¶ The main function of the automated rollup implementation.
This function accepts two context managers, which should return two classes:
TemplateVCS
andProjectVCS
, which represent the cloned template and target project correspondingly.The context managers should prepare the repos, e.g. they should create a temporary directory, clone a repo there, and produce a
TemplateVCS
orProjectVCS
class instance.This function then applies scaraplate rollup of the template to the target project in no-input mode. If the target project contains any changes (as reported by
ProjectVCS.is_dirty()
), they will be committed by callingProjectVCS.commit_changes()
.New in version 0.2.
-
class
scaraplate.automation.base.
TemplateVCS
¶ Bases:
abc.ABC
A base class representing a template retrieved from a VCS (probably residing in a temporary directory).
The resulting directory with template must be within a git repository, see Scaraplate Template for details. But it doesn’t mean that it must be retrieved from git. Template might be retrieved from anywhere, it just has to be in git at the end. That git repo will be used to fill the
TemplateMeta
structure.-
dest_path
¶ Path to the root directory of the template.
-
template_meta
¶ TemplateMeta
filled using the template’s git repo.
-
-
class
scaraplate.automation.base.
ProjectVCS
¶ Bases:
abc.ABC
A base class representing a project retrieved from a VCS (probably residing in a temporary directory).
The project might use any VCS, at this point there’re no assumptions made by scaraplate about the VCS.
-
commit_changes
(template_meta: scaraplate.template.TemplateMeta) → None¶ Commit the changes made to the project. This method is responsible for delivering the changes back to the place the project was retrieved from. For example, if the project is using
git
and it was cloned to a temporary directory, then this method should commit the changes and push them back to git remote.This method will be called only if
ProjectVCS.is_dirty()
has returned True.
-
dest_path
¶ Path to the root directory of the project.
-
is_dirty
() → bool¶ Tell whether the project has any changes not committed to the VCS.
-
-
class
scaraplate.automation.git.
GitCloneTemplateVCS
(template_path: pathlib.Path, template_meta: scaraplate.template.TemplateMeta)¶ Bases:
scaraplate.automation.base.TemplateVCS
A ready to use
TemplateVCS
implementation which:- Uses git
- Clones a git repo with the template to a temporary directory (which is cleaned up afterwards)
- Allows to specify an inner dir inside the git repo as the template root (which is useful for monorepos)
-
classmethod
clone
(clone_url: str, *, clone_ref: Optional[str] = None, monorepo_inner_path: Optional[pathlib.Path] = None) → Iterator[scaraplate.automation.git.GitCloneTemplateVCS]¶ Provides an instance of this class by issuing
git clone
to a tempdir when entering the context manager. Returns a context manager object which after__enter__
returns an instance of this class.Parameters: - clone_url – Any valid
git clone
url. - clone_ref – Git ref to checkout after clone (i.e. branch or tag name).
- monorepo_inner_path – Path to the root dir of template
relative to the root of the repo. If
None
, the root of the repo will be used as the root of template.
- clone_url – Any valid
-
class
scaraplate.automation.git.
GitCloneProjectVCS
(project_path: pathlib.Path, git: scaraplate.automation.git.Git, *, changes_branch: str, commit_author: str, commit_message_template: str)¶ Bases:
scaraplate.automation.base.ProjectVCS
A ready to use
ProjectVCS
implementation which:- Uses git
- Clones a git repo with the project to a temporary directory (which is cleaned up afterwards)
- Allows to specify an inner dir inside the git repo as the project root (which is useful for monorepos)
- Implements
ProjectVCS.commit_changes()
asgit commit
+git push
.
-
classmethod
clone
(clone_url: str, *, clone_ref: Optional[str] = None, monorepo_inner_path: Optional[pathlib.Path] = None, changes_branch: str, commit_author: str, commit_message_template: str = 'Scheduled template update ({update_time:%Y-%m-%d})\n\n* scaraplate version: {scaraplate_version}\n* template commit: {template_meta.commit_url}\n* template ref: {template_meta.head_ref}\n') → Iterator[scaraplate.automation.git.GitCloneProjectVCS]¶ Provides an instance of this class by issuing
git clone
to a tempdir when entering the context manager. Returns a context manager object which after__enter__
returns an instance of this class.Parameters: - clone_url – Any valid
git clone
url. - clone_ref – Git ref to checkout after clone (i.e. branch or tag name).
- monorepo_inner_path – Path to the root dir of project
relative to the root of the repo. If
None
, the root of the repo will be used as the root of project. - changes_branch – The branch name where the changes should be
pushed in the remote. Might be the same as
clone_ref
. Note that this branch is never force-pushed. If upon push the branch already exists in remote and its one-commit diff is different from the one-commit diff of the just created local branch, then the remote branch will be deleted and the local branch will be pushed to replace the previous one. - commit_author – Author name to use for
git commit
, e.g.John Doe <john@example.org>
. - commit_message_template –
str.format()
template which is used to produce a commit message when committing the changes. Available format variables are:update_time
[datetime.datetime
] – the time of updatescaraplate_version
[str
] – scaraplate package versiontemplate_meta
[TemplateMeta
] – template meta returned byTemplateVCS.template_meta()
- clone_url – Any valid
-
class
scaraplate.automation.gitlab.
GitLabCloneTemplateVCS
(git_clone: scaraplate.automation.git.GitCloneTemplateVCS)¶ Bases:
scaraplate.automation.base.TemplateVCS
A class which extends
GitCloneTemplateVCS
with GitLab-specificclone_url
generation.-
classmethod
clone
(project_url: str, private_token: Optional[str] = None, *, clone_ref: Optional[str] = None, monorepo_inner_path: Optional[pathlib.Path] = None) → Iterator[scaraplate.automation.gitlab.GitLabCloneTemplateVCS]¶ Same as
GitCloneTemplateVCS.clone()
except thatclone_url
is replaced withproject_url
andprivate_token
.The
private_token
allows to clone private repos, which are visible only for an authenticated user.Parameters: - project_url – A URL to a GitLab project, e.g.
https://gitlab.example.org/myorganization/myproject
. - private_token – GitLab access token, see https://docs.gitlab.com/ce/api/#oauth2-tokens.
- project_url – A URL to a GitLab project, e.g.
-
classmethod
-
class
scaraplate.automation.gitlab.
GitLabMRProjectVCS
(git_clone: scaraplate.automation.git.GitCloneProjectVCS, *, gitlab_project, mr_title_template: str, mr_description_markdown_template: str)¶ Bases:
scaraplate.automation.base.ProjectVCS
A class which extends
GitCloneProjectVCS
with GitLab-specificclone_url
generation and opens a GitLab Merge Request aftergit push
.-
classmethod
clone
(gitlab_url: str, full_project_name: str, private_token: str, *, mr_title_template: str = 'Scheduled template update ({update_time:%Y-%m-%d})', mr_description_markdown_template: str = '* scaraplate version: `{scaraplate_version}`\n* template commit: {template_meta.commit_url}\n* template ref: {template_meta.head_ref}\n', commit_author: Optional[str] = None, **kwargs) → Iterator[scaraplate.automation.gitlab.GitLabMRProjectVCS]¶ Same as
GitCloneProjectVCS.clone()
with the following exceptions:clone_url
is replaced withgitlab_url
,full_project_name
andprivate_token
.- A GitLab Merge Request (MR) is opened after a successful
git push
.
The
private_token
allows to clone private repos, which are visible only for an authenticated user.As in
GitCloneProjectVCS.clone()
, thechanges_branch
might be the same asclone_ref
. In this case no MR will be opened.A MR will be created only if there’re any changes produced by scaraplate rollup. If a
changes_branch
is already present in remote (i.e. there is a previous automatic rollup which wasn’t merged yet), there’re two possibilities:- If one-commit diffs between the remote’s
changes_branch
and the localchanges_branch
are the same, nothing is done. It means that a MR already exists and it has the same patch as the one which was just produced locally. - If the diffs are different, the remote branch will be deleted, effectively closing the old MR, and a new one will be pushed instead, and a new MR will be opened.
The opened MRs are expected to be merged manually.
Parameters: - gitlab_url – A URL to the GitLab instance, e.g.
https://gitlab.example.org
. - full_project_name – Project name within gitlab, e.g.
myorganization/myproject
. - private_token –
GitLab access token, see https://docs.gitlab.com/ce/api/#oauth2-tokens.
- mr_title_template –
str.format()
template which is used to produce a MR title. Available format variables are:update_time
[datetime.datetime
] – the time of updatetemplate_meta
[TemplateMeta
] – template meta returned byTemplateVCS.template_meta()
- mr_description_markdown_template –
str.format()
template which is used to produce a MR description (which will be rendered as markdown). Available format variables are:update_time
[datetime.datetime
] – the time of updatescaraplate_version
[str
] – scaraplate package versiontemplate_meta
[TemplateMeta
] – template meta returned byTemplateVCS.template_meta()
- commit_author – Author name to use for
git commit
, e.g.John Doe <john@example.org>
. IfNone
, will be retrieved from GitLab as the name of the currently authenticated user (usingprivate_token
).
-
classmethod