Option to provide custom datasources and managers

[fgreinacher]

What would you like Renovate to be able to do?

I as someone running a self-hosted Renovate bot would like to be able to add custom data sources and managers so that I can use Renovate to update dependencies using internal tools with non-standardized formats and APIs.

Describe the solution you'd like to have

The ability to inject custom data sources and managers, probably via some new global config options that specify where the custom logic can be found.

Describe alternatives you've considered

• Managing those dependencies manually, which is cumbersome 😭
• Getting rid of internal tools, which is impossible in our case ⛔️
• Forking Renovate and git patch our changes, which would be egoistic 👾

Additional context

I’d happily contribute this, if it sounds reasonable to you.

[rarkins]

@fgreinacher I think this should now be possible Both manager and datasource implementation can be wholly contained and dynamically detected now within lib/manager/* and lib/datasource/* respectively. I would be interested to hear if this approach works for you.

[dominykas]

Would it be possible to also have a config param which would point to a folder to load additional managers?

Use cases:

• to add a manager right now one has to open a PR here, wait for the release, get an agreement - this is not always efficient
• to develop a manager, one has to fork renovate and use the fork - it would be nice to be able to test the managers independently
• it does not necessarily make sense to have it in renovate for everyone when various specific internal formats are used, so a pluggable manager option could solve that case

[SirUli]

So from what i understand it is "out of the box" currently not possible to configure a custom datasource directly from renovate but through a custom development located in `lib/datasource/*``, correct? I worry less about the managers since the regex manager is quite powerful.

E.g. I do have an Bot that uploads releases from a private vendor website to a common repository. My idea was that renovate calls a specific API or extracts the data from a file to basically get the latest version of a product. Currently - as i understand it - there is no such generic datasource, correct? I would have to build that myself and - to be able to ship it properly - would have to create a PR here.

[rarkins]

So from what i understand it is "out of the box" currently not possible to configure a custom datasource directly from renovate but through a custom development located in `lib/datasource/*``, correct?

Correct.

Currently - as i understand it - there is no such generic datasource, correct?

Correct. We're open to ideas about this though, e.g. in the same spirit as the regex manager but not necessarily using regex. I was looking into ideas like using JSONPath to transform results, with a few generic parsers supported (JSON, XML, TOML, YAML, etc).

[SirUli]

Thanks for the confirmation, was reading code for a while since many of these datasources are totally unknown to me. https://docs.renovatebot.com/modules/datasource/ could use some links and a description to information of each of the datasources. I guess it makes sense to send a pull request with respective readmes for such a purpose.

So my thoughts on the generic datasource:

• Source could be either an HTTP(S) or a local endpoint. HTTP(S) could require authentication and parameters whereas the local file might only need a path
• Agree with the point on parsers - additionally a regex parser might be of use
• Source might deliver many different versions, so it might need a "versioning" hint

I'm not too much an expert in renovate so far (although it worked pretty quickly i have to say!), so bear with me on the next lines.

When looking at the file where i would want to exchange the version, i could envision this (extracted through the regexmanager):

# renovate: datasource=custom datasourceName=nexus depName=somesoftware versioning=loose
somesoftware_version: "56"

and in my config for renovate i would have a datasource configuration:

{
  "customDatasource": [
    {
      "name": "nexus",
      "path": "https://nexus.example.com/service/rest/v1/search?group=&repository=myRepo&q={{ filename }}",
      "auth": "basic",
      "authdata": "foo:bar"
      "matcher": "jsonpath"
      "jsonpath": "$.[*].name"
      "match": ^{{ filename }}_(?P<ver>\d+)\.rpm$
    }
  ]
}

Maybe that helps to get a discussion going?

[dominykas]

Is there reluctance to add a full fledged manager plug-in support?

[rarkins]

As far as I'm aware, there's nothing stopping you "dropping in" managers today if you then run the build command from source yourself. I'm not ready to commit to any run-time plugin architecture partly because I think we'll want to bundle Renovate into a single file to improve load time in the app.

[dominykas]

As far as I'm aware, there's nothing stopping you "dropping in" managers today

But that means you need to basically clone renovate and run it from git? Or monkey-patch node_modules where it is installed?

I'm not ready to commit to any run-time plugin architecture

I feel the sentiment - fair enough, but I do have to point out that the existing managers have a pretty nice API surface and adding new managers is not that hard, so good job there and I do think it's versatile enough going forward, should you chose to stick with that :)

[rarkins]

But that means you need to basically clone renovate and run it from git?

No, I'd more anticipate somebody having an automated build process that takes Renovate, adds in additional files, then rebuilds and publishes it to some internal registry or Docker image.

The resistance is mostly about unnecessarily painting ourselves into a corner. We've rearchitected our internal structures a lot and although happy with it for now, I don't know if in future we'll need more changes again, which would be made really hard if we've essentially "externalized" that API by making it officially in support of plugins. None of the maintainers have a need for plugins so it would require someone who does to step forward in the project and make us feel confident we're not getting saddled with technical debt that people expect us to maintain indefinitely.

[pandorasNox]

My feeling would be to solve this problem with external REST api endpoints (or grpc, or graphql), which means external servers for one or multiple datasources/managers (which e.g. you could write and deploy for yourself just following a given REST api scheme) which renovates understands and in the bot config one can just point to a list of reachable endpoints (or a list of endpoint configuration files / directory, since e.g. tls support and possible more is something one would want to add)

As an example I think of something like in kubernetes you have admission controller, mutating and validating webhooks for example

So like a datasource webhook

[okainov]

As another examples of possible "custom" datasource might be:

• Webservers with a list of files.

For us we use it for cmake and maven, for example:

CMAKE_VERSION=3.17
CMAKE_BUILD=3 # Download, extract and install cmake 
cd /tmp 
wget -q https://cmake.org/files/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.${CMAKE_BUILD}.tar.gz
...

MAVEN_VERSION=3.6.3
mkdir -p /opt && cd /opt
wget -q https://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz
tar -xzf apache-maven-$MAVEN_VERSION-bin.tar.gz
rm -rf apache-maven-$MAVEN_VERSION-bin.tar.gz
ln -s apache-maven-$MAVEN_VERSION maven

I see today this can be somehow workarounded by using some repology repo (at least I found ones for Maven and Cmake), but in general it might point to internal webserver with internal builds, so it would be still useful

• Files on the file system / file share

For some libs we get them from a file share. Theoretically it can be turned into a webserver, but default flow would be to check the file properties, for example

[jamietanna]

Closing as we've supported these for a few years now 🚀 Feel free to raise fresh Discussion(s) if there are additional features to add