kopia lustrzana https://github.com/jupyterhub/repo2docker
Porównaj commity
9 Commity
3f9b17810c
...
63ae629117
Autor | SHA1 | Data |
---|---|---|
Steve Purves | 63ae629117 | |
Erik Sundell | d2f467cef6 | |
pre-commit-ci[bot] | 0e7907f86b | |
Simon Li | 243669b928 | |
Simon Li | d4bb9c14da | |
stevejpurves | 4ae375e6ae | |
stevejpurves | 359e0465a9 | |
Steve Purves | 9e0bd86a87 | |
stevejpurves | e85495cf04 |
|
@ -11,7 +11,7 @@
|
|||
repos:
|
||||
# Autoformat: Python code, syntax patterns are modernized
|
||||
- repo: https://github.com/asottile/pyupgrade
|
||||
rev: v3.15.0
|
||||
rev: v3.15.2
|
||||
hooks:
|
||||
- id: pyupgrade
|
||||
args:
|
||||
|
@ -23,7 +23,7 @@ repos:
|
|||
|
||||
# Autoformat: Python code
|
||||
- repo: https://github.com/psf/black
|
||||
rev: 24.1.1
|
||||
rev: 24.3.0
|
||||
hooks:
|
||||
- id: black
|
||||
args:
|
||||
|
|
|
@ -1,5 +1,63 @@
|
|||
# Changelog
|
||||
|
||||
## Version 2024.03.0
|
||||
|
||||
([full changelog](https://github.com/jupyterhub/repo2docker/compare/2023.06.0...2024.01.03))
|
||||
|
||||
### New features added
|
||||
|
||||
- Implement support for dockerignore and containerignore [#1205](https://github.com/jupyterhub/repo2docker/pull/1205) ([@sgaist](https://github.com/sgaist))
|
||||
|
||||
### Enhancements made
|
||||
|
||||
- rstudio: log-level info to stderr [#1317](https://github.com/jupyterhub/repo2docker/pull/1317) ([@manics](https://github.com/manics))
|
||||
|
||||
### Bugs fixed
|
||||
|
||||
- Get Zenodo working again [#1315](https://github.com/jupyterhub/repo2docker/pull/1315) ([@manics](https://github.com/manics))
|
||||
|
||||
### Maintenance and upkeep improvements
|
||||
|
||||
- Update mamba to 1.5.7 from 1.5.1, and conda to 24.3.0 from 23.7.4 [#1337](https://github.com/jupyterhub/repo2docker/pull/1337) ([@SylvainCorlay](https://github.com/SylvainCorlay))
|
||||
- Dockerfile: bump alpine from 3.17 to 3.19 and Python 3.10 to 3.11 [#1332](https://github.com/jupyterhub/repo2docker/pull/1332) ([@yuvipanda](https://github.com/yuvipanda))
|
||||
- Upgrade base image from to Ubuntu 22.04 from 18.04 [#1287](https://github.com/jupyterhub/repo2docker/pull/1287) ([@yuvipanda](https://github.com/yuvipanda))
|
||||
|
||||
### Documentation improvements
|
||||
|
||||
- Add Ubuntu 22.04 upgrade guide [#1309](https://github.com/jupyterhub/repo2docker/pull/1309) ([@manics](https://github.com/manics))
|
||||
- Update version of R available [#1288](https://github.com/jupyterhub/repo2docker/pull/1288) ([@yuvipanda](https://github.com/yuvipanda))
|
||||
- Add changelog for 2023.06.0 [#1286](https://github.com/jupyterhub/repo2docker/pull/1286) ([@yuvipanda](https://github.com/yuvipanda))
|
||||
|
||||
### Other merged PRs
|
||||
|
||||
- [pre-commit.ci] pre-commit autoupdate [#1333](https://github.com/jupyterhub/repo2docker/pull/1333) ([@pre-commit-ci](https://github.com/pre-commit-ci))
|
||||
- build(deps): bump codecov/codecov-action from 3 to 4 [#1331](https://github.com/jupyterhub/repo2docker/pull/1331) ([@dependabot](https://github.com/dependabot))
|
||||
- Support pytest=8 [#1330](https://github.com/jupyterhub/repo2docker/pull/1330) ([@manics](https://github.com/manics))
|
||||
- Update versioneer [#1329](https://github.com/jupyterhub/repo2docker/pull/1329) ([@TimoRoth](https://github.com/TimoRoth))
|
||||
- build(deps): bump actions/setup-python from 4 to 5 [#1328](https://github.com/jupyterhub/repo2docker/pull/1328) ([@dependabot](https://github.com/dependabot))
|
||||
- build(deps): bump actions/upload-artifact from 3 to 4 [#1327](https://github.com/jupyterhub/repo2docker/pull/1327) ([@dependabot](https://github.com/dependabot))
|
||||
- Add NYCU Dataverse [#1326](https://github.com/jupyterhub/repo2docker/pull/1326) ([@twtw](https://github.com/twtw))
|
||||
- [pre-commit.ci] pre-commit autoupdate [#1322](https://github.com/jupyterhub/repo2docker/pull/1322) ([@pre-commit-ci](https://github.com/pre-commit-ci))
|
||||
- New domain for Edmond MPG repository [#1321](https://github.com/jupyterhub/repo2docker/pull/1321) ([@haarli](https://github.com/haarli))
|
||||
- [pre-commit.ci] pre-commit autoupdate [#1319](https://github.com/jupyterhub/repo2docker/pull/1319) ([@pre-commit-ci](https://github.com/pre-commit-ci))
|
||||
- [MRG] docs: Add base_image parameter example. [#1318](https://github.com/jupyterhub/repo2docker/pull/1318) ([@hiroyuki-sato](https://github.com/hiroyuki-sato))
|
||||
- Upgrade mamba and refreeze [#1313](https://github.com/jupyterhub/repo2docker/pull/1313) ([@manics](https://github.com/manics))
|
||||
- r: Bump version of rsession-proxy [#1310](https://github.com/jupyterhub/repo2docker/pull/1310) ([@yuvipanda](https://github.com/yuvipanda))
|
||||
- build(deps): bump actions/checkout from 3 to 4 [#1308](https://github.com/jupyterhub/repo2docker/pull/1308) ([@dependabot](https://github.com/dependabot))
|
||||
- build(deps): bump docker/build-push-action from 4 to 5 [#1307](https://github.com/jupyterhub/repo2docker/pull/1307) ([@dependabot](https://github.com/dependabot))
|
||||
- build(deps): bump docker/setup-qemu-action from 2 to 3 [#1306](https://github.com/jupyterhub/repo2docker/pull/1306) ([@dependabot](https://github.com/dependabot))
|
||||
- build(deps): bump docker/setup-buildx-action from 2 to 3 [#1305](https://github.com/jupyterhub/repo2docker/pull/1305) ([@dependabot](https://github.com/dependabot))
|
||||
- Update conda and mamba [#1299](https://github.com/jupyterhub/repo2docker/pull/1299) ([@SylvainCorlay](https://github.com/SylvainCorlay))
|
||||
- Point to official documentation for handling JupyterLab workspace [#1294](https://github.com/jupyterhub/repo2docker/pull/1294) ([@fcollonval](https://github.com/fcollonval))
|
||||
- Fix rstudio-build selection [#1293](https://github.com/jupyterhub/repo2docker/pull/1293) ([@yamaton](https://github.com/yamaton))
|
||||
- [pre-commit.ci] pre-commit autoupdate [#1291](https://github.com/jupyterhub/repo2docker/pull/1291) ([@pre-commit-ci](https://github.com/pre-commit-ci))
|
||||
|
||||
### Contributors to this release
|
||||
|
||||
([GitHub contributors page for this release](https://github.com/jupyterhub/repo2docker/graphs/contributors?from=2023-06-13&to=2024-03-28&type=c))
|
||||
|
||||
[@annakrystalli](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Aannakrystalli+updated%3A2023-06-13..2024-03-28&type=Issues) | [@betatim](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Abetatim+updated%3A2023-06-13..2024-03-28&type=Issues) | [@bollwyvl](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Abollwyvl+updated%3A2023-06-13..2024-03-28&type=Issues) | [@consideRatio](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3AconsideRatio+updated%3A2023-06-13..2024-03-28&type=Issues) | [@dependabot](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Adependabot+updated%3A2023-06-13..2024-03-28&type=Issues) | [@dolfinus](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Adolfinus+updated%3A2023-06-13..2024-03-28&type=Issues) | [@fcollonval](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Afcollonval+updated%3A2023-06-13..2024-03-28&type=Issues) | [@felder](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Afelder+updated%3A2023-06-13..2024-03-28&type=Issues) | [@haarli](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ahaarli+updated%3A2023-06-13..2024-03-28&type=Issues) | [@hiroyuki-sato](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ahiroyuki-sato+updated%3A2023-06-13..2024-03-28&type=Issues) | [@manics](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Amanics+updated%3A2023-06-13..2024-03-28&type=Issues) | [@mathieuboudreau](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Amathieuboudreau+updated%3A2023-06-13..2024-03-28&type=Issues) | [@minrk](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Aminrk+updated%3A2023-06-13..2024-03-28&type=Issues) | [@pre-commit-ci](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Apre-commit-ci+updated%3A2023-06-13..2024-03-28&type=Issues) | [@rgaiacs](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Argaiacs+updated%3A2023-06-13..2024-03-28&type=Issues) | [@ryanlovett](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Aryanlovett+updated%3A2023-06-13..2024-03-28&type=Issues) | [@sgaist](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Asgaist+updated%3A2023-06-13..2024-03-28&type=Issues) | [@SylvainCorlay](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3ASylvainCorlay+updated%3A2023-06-13..2024-03-28&type=Issues) | [@TimoRoth](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3ATimoRoth+updated%3A2023-06-13..2024-03-28&type=Issues) | [@twtw](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Atwtw+updated%3A2023-06-13..2024-03-28&type=Issues) | [@welcome](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Awelcome+updated%3A2023-06-13..2024-03-28&type=Issues) | [@yamaton](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ayamaton+updated%3A2023-06-13..2024-03-28&type=Issues) | [@yuvipanda](https://github.com/search?q=repo%3Ajupyterhub%2Frepo2docker+involves%3Ayuvipanda+updated%3A2023-06-13..2024-03-28&type=Issues)
|
||||
|
||||
## Version 2023.06.0
|
||||
|
||||
([full changelog](https://github.com/jupyterhub/repo2docker/compare/2022.10.0...2023.06.0))
|
||||
|
|
|
@ -154,6 +154,7 @@ class Repo2Docker(Application):
|
|||
contentproviders.Swhid,
|
||||
contentproviders.Mercurial,
|
||||
contentproviders.Git,
|
||||
contentproviders.Meca,
|
||||
],
|
||||
config=True,
|
||||
help="""
|
||||
|
|
|
@ -3,6 +3,7 @@ from .dataverse import Dataverse
|
|||
from .figshare import Figshare
|
||||
from .git import Git
|
||||
from .hydroshare import Hydroshare
|
||||
from .meca import Meca
|
||||
from .mercurial import Mercurial
|
||||
from .swhid import Swhid
|
||||
from .zenodo import Zenodo
|
||||
|
|
|
@ -0,0 +1,130 @@
|
|||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import xml.etree.ElementTree as ET
|
||||
from hashlib import md5
|
||||
from os import path
|
||||
from urllib.parse import urlparse, urlunparse
|
||||
from zipfile import ZipFile, is_zipfile
|
||||
|
||||
from requests import Session
|
||||
|
||||
from .base import ContentProvider
|
||||
|
||||
|
||||
def get_hashed_slug(url, changes_with_content):
|
||||
"""Returns a unique slug that is invariant to query parameters in the url"""
|
||||
parsed_url = urlparse(url)
|
||||
stripped_url = urlunparse(
|
||||
(parsed_url.scheme, parsed_url.netloc, parsed_url.path, "", "", "")
|
||||
)
|
||||
|
||||
return "meca-" + md5(f"{stripped_url}-{changes_with_content}".encode()).hexdigest()
|
||||
|
||||
|
||||
def fetch_zipfile(session, url, dst_dir):
|
||||
resp = session.get(url, headers={"accept": "application/zip"}, stream=True)
|
||||
resp.raise_for_status()
|
||||
|
||||
dst_filename = path.join(dst_dir, "meca.zip")
|
||||
with open(dst_filename, "wb") as dst:
|
||||
for chunk in resp.iter_content(chunk_size=128):
|
||||
dst.write(chunk)
|
||||
|
||||
return dst_filename
|
||||
|
||||
|
||||
def extract_validate_and_identify_bundle(zip_filename, dst_dir):
|
||||
if not os.path.exists(zip_filename):
|
||||
raise RuntimeError("Downloaded MECA bundle not found")
|
||||
|
||||
if not is_zipfile(zip_filename):
|
||||
raise RuntimeError("MECA bundle is not a zip file")
|
||||
|
||||
with ZipFile(zip_filename, "r") as zip_ref:
|
||||
zip_ref.extractall(dst_dir)
|
||||
|
||||
try:
|
||||
manifest = path.join(dst_dir, "manifest.xml")
|
||||
if not os.path.exists(manifest):
|
||||
raise RuntimeError("MECA bundle is missing manifest.xml")
|
||||
article_source_dir = "bundle/"
|
||||
|
||||
tree = ET.parse(manifest)
|
||||
root = tree.getroot()
|
||||
|
||||
bundle_instance = root.findall(
|
||||
"{*}item[@item-type='article-source-directory']/{*}instance"
|
||||
)
|
||||
for attr in bundle_instance[0].attrib:
|
||||
if attr.endswith("href"):
|
||||
article_source_dir = bundle_instance[0].get(attr)
|
||||
|
||||
return True, path.join(dst_dir, article_source_dir)
|
||||
except:
|
||||
return False, dst_dir
|
||||
|
||||
|
||||
class Meca(ContentProvider):
|
||||
"""A repo2docker content provider for MECA bundles"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.session = Session()
|
||||
self.session.headers.update(
|
||||
{
|
||||
"user-agent": f"repo2docker MECA",
|
||||
}
|
||||
)
|
||||
|
||||
def detect(self, spec, ref=None, extra_args=None):
|
||||
"""`spec` contains a faux protocol of meca+http[s] for detection purposes
|
||||
and we assume `spec` trusted as a reachable MECA bundle from an allowed origin
|
||||
(binderhub RepoProvider class is already checking for this).
|
||||
|
||||
An other HEAD check in made here in order to get the content-length header
|
||||
"""
|
||||
parsed = urlparse(spec)
|
||||
if not parsed.scheme.endswith("+meca"):
|
||||
return None
|
||||
parsed = parsed._replace(scheme=parsed.scheme[:-5])
|
||||
url = urlunparse(parsed)
|
||||
|
||||
headers = self.session.head(url).headers
|
||||
changes_with_content = headers.get("ETag") or headers.get("Content-Length")
|
||||
|
||||
self.hashed_slug = get_hashed_slug(url, changes_with_content)
|
||||
|
||||
return {"url": url, "slug": self.hashed_slug}
|
||||
|
||||
def fetch(self, spec, output_dir, yield_output=False):
|
||||
hashed_slug = spec["slug"]
|
||||
url = spec["url"]
|
||||
|
||||
yield f"Creating temporary directory.\n"
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
yield f"Temporary directory created at {tmpdir}.\n"
|
||||
|
||||
yield f"Fetching MECA Bundle {url}.\n"
|
||||
zip_filename = fetch_zipfile(self.session, url, tmpdir)
|
||||
|
||||
yield f"Extracting MECA Bundle {zip_filename}.\n"
|
||||
is_meca, bundle_dir = extract_validate_and_identify_bundle(
|
||||
zip_filename, tmpdir
|
||||
)
|
||||
|
||||
if not is_meca:
|
||||
yield f"This doesn't look like a meca bundle, extracting everything.\n"
|
||||
|
||||
yield f"Copying MECA Bundle at {bundle_dir} to {output_dir}.\n"
|
||||
files = os.listdir(bundle_dir)
|
||||
for f in files:
|
||||
shutil.move(os.path.join(bundle_dir, f), output_dir)
|
||||
|
||||
yield f"Removing temporary directory.\n"
|
||||
|
||||
yield f"MECA Bundle {hashed_slug} fetched and unpacked.\n"
|
||||
|
||||
@property
|
||||
def content_id(self):
|
||||
return self.hashed_slug
|
Ładowanie…
Reference in New Issue