feat(http): Switch to memory cache provider

[zharinov]

Changes

Make caching of GET/HEAD HTTP requests disabled by default.
When it's needed, the memCacheProvider is be used.

Context

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[zharinov]

• Blocked by feat(http): Create memory cache provider #33900

[rarkins]

I will test first

[rarkins]

Can you explain why the memCache=false parts are no longer needed?

[zharinov]

memCache=false is the equivalent of the absent cacheProvider option

[zharinov]

And yes, no memory cache is the default now

[rarkins]

How can we be confident that we haven't missed anything?

[zharinov]

Generally speaking, we can't. What we can is analyze HTTP stats and watch for repeating GET requests (sometimes, though, it's intended).

[rarkins]

Can you summarize all the cases where http memcache is still being explicitly enabled in this PR (seems to be within platform code) and we evaluate whether we're better to add platform-level caching (e.g. of PR lists) instead of relying on http-level? Maybe you can do this through inline PR comments

[rarkins]

This should have a significant reduction in the number of http requests which we cache in memory. This should apply especially to datasource requests

[rarkins]

@zharinov can we log when caching fails/falls through to the lowest level?

[rarkins]

Also needs deconflicting

[rarkins]

I switched this to logger.error and ran on containerbase/base with dryRun=lookup and got many errors. They're all either /v2/ queries or /token? ones from docker datasource.

Maybe this datasource should do its own caching of these, with its own TTLs?

[rarkins]

I think I'd prefer to accumulate these URLs into a Set and then at the end print a debug message about number of URLs using legacy caching. Then we can capture and graph that in our hosted app and know when we reach 0

[rarkins]

Needs a refactor of docker datasource too

[Bruno-bot-stack]

Leave a review

[rarkins]

When I run on a fork of the renovate repo, and there are PRs needing changelogs, we see some more cache fallbacks:

DEBUG: Cache hit URLs (repository=renovate-reproductions/renovate2)
       "count": 7,
       "hits": {
         "https://api.github.com/repos/yarnpkg/berry": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 1
         },
         "https://api.github.com/repos/yarnpkg/berry/git/trees/master?recursive=1": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 1
         },
         "https://api.github.com/repos/yarnpkg/berry/git/blobs/3e96b54ffe13ba43b83ee1844a3ba37aeb4f77f8": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 1
         },
         "https://api.github.com/repos/aws/aws-sdk-js-v3": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 5
         },
         "https://api.github.com/repos/aws/aws-sdk-js-v3/git/trees/main?recursive=1": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 5
         },
         "https://api.github.com/repos/eslint/eslint": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 1
         },
         "https://api.github.com/repos/eslint/eslint/git/blobs/7d070fd0c3cbfeb53f719f85ee0e6677b70a0f5b": {
           "callsite": "async Object.getReleaseNotesMd (/Users/rhys/src/renovatebot/renovate/lib/workers/repository/update/pr/changelog/github/index.ts)",
           "count": 1
         }
       }