Improve changelog detection for monorepos

[rarkins]

The original data model for changelog was to assume a 1:1 mapping between package and source repository.

For monorepos, there are multiple challenges:

• Nested changelog.md files (Support nested changelog.md files #2162)
• GitHub releases are sometimes done using raw version (e.g. v6.0.0) and sometimes with name+version, e.g. [email protected]

So our challenge is:

1. Find the right GitHub Release or changelog file for the package
2. De-duplicate them in PR bodies

Also related: what to do if a changelog.md file is updated (e.g. improved) in master branch well after publication of the package? Most of the time we'd want to update the PRs with that new info, but perhaps not always.

• If it is a github repo and then look for a "Release" named [email protected], [email protected], v1.0.0 and 1.0.0
• If there is a changelog file in the root of the repo, use it
• If there is a file like ........./package/changelog.md, then use it (i.e. changelog nested)

Then the challenge is how to de-duplicate it within the PR body. If each package has identical from/to versions and identical content, then we should just display it once. But what if it's a mix of shared and non-shared?

[rarkins]

In they the release notes for manager-package-release should always be the same. But the raw commit "compare" link can be different, namely whether the user is comparing it to the previous stable version or the previous unstable version.

Each "release" should have a "gitRef", whether that's a tag (e.g. v1.0.1) or a raw commit hash. Then the exact compare links can be generated per-PR.

[Flydiverny]

Having just found this issue after trying to get changelogs working for our monorepo packages, I take it there's no workaround at the moment? Using lerna with individual versioning and have one CHANGELOG.md in each package and also tried to update the release tags (@scope/[email protected]) with changelog entries with no success. Found it curious that changelogs wouldn't be picked up while compare links are using the correct tags and versions as well as showing all the links for each release.

[rarkins]

Can you replicate it in a public repo or find a public lerna repo that uses the same approach you do, that can be tested against?

[Flydiverny]

Not sure if this is something thats interesting or not but figured I'd share it here:
https://github.com/npm/rfcs/blob/latest/implemented/0010-monorepo-subdirectory-declaration.md

Suppose this could potentially help with detecting changelog files for monorepo packages? :)

[rarkins]

Thanks, I've seen that one and partially implemented it already. We already do a search for subdirectories containing the package name so that captures nearly all the use cases already anyway.

[felixfbecker]

Even if a changelog could not be detected, it would be nice if Renovate could add a "Compare source" link that links to comparison page of that subfolder so I can inspect the commits

[rarkins]

@felixfbecker I've never seen a subfolder compare link before. Do you have an example for GitHub?

[felixfbecker]

Hmm, doesn't seem to exit in the UI. Maybe a link to the history page of the folder then? https://github.com/renovatebot/renovate/commits/master/docs

[rarkins]

OK so if source directory is defined in package.json but no CHANGELOG file is found within that directory, then link to the directory's commit history?

[Flydiverny]

Thanks, I've seen that one and partially implemented it already. We already do a search for subdirectories containing the package name so that captures nearly all the use cases already anyway.

Does this mean that if I have a monorepo with a package name banana
and these files

- ..stuff in root..
- packages/banana/package.json
- packages/banana/CHANGELOG.md

It would detect the changelog and present it already?

I suppose this doesn't work for scoped packages where I would have eg @flydiverny/banana as package name but same file structure as above.

Is there anything I can set in package.json files to make it detect nested changelogs today?

[rarkins]

Can you describe your situation in full? Eg self hosted or GitHub.com? Directory structure, etc

[Flydiverny]

Its on github. So using this as an example:
monorepo with lerna + yarn

packages/renovate-bug-dep contains a package with name @flydiverny/renovate-bug-dep.
There I have a nested CHANGELOG.md.

For version 1.1.4 of the package there's also a github release tagged as @flydiverny/[email protected]

Here's a PR from renovate trying to update this dependency: Flydiverny/renovate-workspace#2

Don't mind the repo and package names :)

In our project its similar with the main difference being private github repo and private packages on npm.

[rarkins]

@Flydiverny thanks for the example! That's exactly what I needed. Can I assume that the GitHub Releases entry and CHANGELOG.md entries should always be identical, i.e. we can use whichever one works best for our internal logic?

[Flydiverny]

@rarkins Yes they are supposed to match :) using lerna to generate both of them here as they recently added a --github-release flag.

The github release requires the end user to specify github token tho so its a bit more of a hassle to use. But I personally prefer github releases anyway.

[rarkins]

Alright thanks, I think we have enough info to fix it!

[rarkins]

Next steps: debug changelog retrieval using the reproduction repos from @Flydiverny here: #2926 (comment)

Either fix/improve our existing release parsing (we should be able to locate the package/release combination) or locate the nested CHANGELOG.md files

[koba04]

Is there any update on this?
I have the same issue.

We use yarn workspaces + lerna for a monorepo and publish a new package using lerna publish.
As a result, the CHANGELOG.md is placed in each package and doesn't exist in the root directory on the repository.

Should I put CHANGELOG.md that aggregates all changelog for packages in the root directory?
Or should I put the changes in the GitHub releases?

The following is the monorepo.
https://github.com/kintone/js-sdk

The following is a PR to update the package version by Renovate.
koba04/gatsby-source-kintone#13

Thanks!

[rarkins]

@koba04 No, you shouldn't need to change the way your do the individual CHANGELOG files, Renovate should be updated to accommodate/detect that.

[rarkins]

@koba04 actually, you do need to make some changes though. Currently your monorepo is not complying with npm's package.json recommendation:

Notably here: https://github.com/kintone/js-sdk/blob/edc5d0cd75e577b90d22c0234bbbcfaa5263a946/packages/rest-api-client/package.json#L37-L39

[viceice]

Always seeing duplicate version entries for traefik

[koba04]

@rarkins
As you mentioned at #2926 (comment), I've added the repostitory.directory property in a package in monorepo at kintone/js-sdk#29, but Renovate doesn't recognize CHANGELOG.md in the packages/*** directory.

koba04/gatsby-source-kintone#28

Am I missing something?

[rarkins]

@koba04 can create a minimal reproduction to debug against?

[koba04]

@rarkins I've created a reproduction.

• PR: Update dependency @koba04/lerna-yarn-sandbox-package-a to ^1.5.2 - abandoned koba04/renovate-sandbox#15
• CHANGELOG: https://github.com/koba04/lerna-yarn-sandbox/blob/master/packages/lerna-yarn-sandbox-package-a/CHANGELOG.md

I've expected that the PR includes the changes in the CHANGELOG.md

[github-actions]

Thank you for providing a reproduction.
The label will be updated to reproduction:confirmed once someone else has confirmed they can reproduce the problem with it.

[k-yle]

Hi @rarkins, I had a go at fixing this but ran out of time. I think the issue are:

1. this API call needs ?recursive=1 to return all files, not just the root level one; and
2. the regex from changelog-filename-regex uses ^ and $ so it doesn't work if the file name is packages/x/CHANGELOG.md

but, this approach still fails if the package doesn't specify repository.directory in package.json.

So perhaps #9061 is a better approach?

[github-actions]

Thank you for providing a reproduction! 🎉 🚀

The Renovate team will take a look at the reproduction repository. Once we confirm the provided repository reproduces the problem, the label will be changed to reproduction:confirmed.

[renovate-release]

🎉 This issue has been resolved in version 25.47.6 🎉

The release is available on:

• GitHub release
• 25.47.6

Your semantic-release bot 📦🚀