Colour handling improvements #8702

Jermolene · 2024-10-25T09:02:10Z

Introduction

This PR brings several new features for end users:

Automatically switching between a dark and light palette as the operating system setting changes (and to do so without making the wiki dirty)
Customisation options for palettes. For example, users might choose a base hue, with the colours of the palette automatically adapting to it
A generalisation of the dark vs. light mechanism to allow an arbitrary number of distinct schemes that are dynamically selected. For example, a palette that has a different scheme for night, morning, day and evening that automatically change with the time of day

There are also new capabilities for palette authors:

Inheritance for palettes, making it easy to create chains of variants of a base palette
Self contained palettes that can contain both dark and light variants (or variants for any other custom scheme)

To make all of these new features possible, this PR also includes some useful new general purpose mechanisms and features:

Background actions that are triggered whenever there is a change to the results of a specified filter
Several new filter operators for manipulating colour values. The underlying functionality comes from the color.js library
New media query tracking mechanism that can track the results of any CSS media query (not just dark mode), storing the results in a shadow $:/info/... tiddler
New changecount filter operator
New :apply filter run prefix

Try the Demo

A preview build is available at https://deploy-preview-8702--tiddlywiki-previews.netlify.app/

Changes to Palettes

Palette entries are now defined as filters that must be evaluated, rather than wikitext that must be wikified.

This makes it possible to create palettes that reference and modify other colours. For example:

tiddler-controls-foreground-selected: [tf.interpolate-colours[background],[foreground],[0.9]]

Retrospectively re-interpreting palettes entries as filters would normally mean that <<colour background>> would no longer work. Instead these entries are automatically converted to the updated form [function[colour],[background]].

Palette Compilation

The key idea underpinning these changes is a fundamental change to the way that TiddlyWiki handles palettes. At the moment, palette entries are named items that can contain either a CSS colour string, a CSS colour token like "inherit", or can use the <<colour>> macro to reference another colour palette entry. Thus, palette entries have to be wikified before they can be used. This has turned out to be very limiting and doesn't provide a viable path to the complex colour manipulations shown above. Switching to filters might make things worse, by encouraging authors to use complex expressions within palettes.

The fix is compiled palettes: at the point of switching to a new palette, the colours within it are "compiled" to raw CSS colour values (typically but not necessarily in #rrggbbaa format). This allows palette entries to be used directly, without the requirement to wikify them.

The static palette is created in a new system tiddler $:/temp/palette-colours by an action procedure that is invoked at startup and when switching to a new palette.

There should not be any backwards compatibility issues because the use of background actions means that any code that changes $:/palette will automatically trigger the recompilation of the palette.

This change also allows us to change the <<colour>> procedure to be a function, which allows it to be used as the value for a style attribute:

<div style.background=<<colour tiddler-background>>>

Automatic Palette Readability Tests

Palettes can opt to include readability tests as special palette entries. The results of these tests are shown at the bottom of the palette chooser. For example:

?base-background-ink: [tf.check-colour-contrast[base-background],[base-ink],[45]]

The test framework looks for palette entries starting with a question mark and runs the associated filters. The filter should return nothing if there are no errors, or a textual error message if the conditions are violated. Sample output:

alert-contrast: 42.357: alert-background/foreground contrast is too low
background-foreground-contrast: 41.915: background/foreground contrast is too low
base-ink-secondary: 42.357: base-ink/base-secondary contrast is too low
base-paper-ink: 41.915: base-paper/base-ink contrast is too low
base-paper-primary: 32.682: base-paper/base-primary contrast is too low
base-paper-tertiary: 30.502: base-paper/base-tertiary contrast is too low
code-contrast: 28.565: code-background/code-foreground contrast is too low

Colour Manipulation

We need a colour manipulation library that can calculate variants of colours. Only color.js met the requirements of being able to work with P3 colours and the OKLCH colour space. It also includes a CSS colour string parser which can replace the simple one that TiddlyWiki has always incorporated.

Media Query Tracker

The CSS media query tracker allows a media query to be bound to an info tiddler so that the current state of the query is reflected in the value of the tiddler. The value is updated dynamically.

The use of the info mechanism for the CSS media query tracker means that these tiddlers are dynamically created as shadow tiddlers within the $:/temp/info plugin, and so do not appear in tiddler lists.

The mechanism is used to implement the existing dark mode tiddler with the following configuration:

title: $:/core/wiki/config/MediaQueryTrackers/DarkLightSwitch
tags: $:/tags/MediaQueryTracker
media-query: (prefers-color-scheme: dark)
info-tiddler: $:/info/browser/darkmode
info-tiddler-alt: $:/info/darkmode

Note the use of info-tiddler-alt to specify a redundant secondary info tiddler. This is used by the dark mode tracker to maintain compatibility while changing the info tiddler title for consistency.

Background Actions

The new background actions mechanism allows action strings to be invoked automatically in the background whenever the results of a filter change.

The preview includes a demonstration background action that displays an alert with a list of tiddlers in the story river whenever the story river changes:

title: SampleBackgroundAction: Story Change
tags: $:/tags/BackgroundAction
track-filter: [list[$:/StoryList]]

<$action-sendmessage $message="tm-notify" $param="SampleBackgroundAction: Story Change" list={{$:/StoryList!!list}}/>

Story List:

<ol>
<$list filter="[enlist<list>]">
<li>
<$text text=<<currentTiddler>>/>
</li>
</$list>
</ol>

`apply` Filter Run Prefix

An experimental new filter run prefix that makes it possible to use computed values as variables within a filter run. For example:

\function tf.interpolate-colours(paletteEntryA,paletteEntryB,weight)
[function[colour],<paletteEntryA>] [function[colour],<paletteEntryB>] :apply[<weight>colour-interpolate:oklch<$1>,<$2>]
\end tf.interpolate-colours

Backwards Compatibility

The current content of $:/PaletteManager is moved into $:/PaletteEditor, and $:/PaletteManager repurposed as the control panel palette switcher
$:/config/DefaultColourMappings/ now only supports CSS colours, and not indirections via <<colour X>> or [function[colour],[X]]

References

The media query tracker and background actions mechanisms were originally developed in a separate pull request
[IDEA] Tools for manipulating colours and palettes #5236 discusses the general idea of dynamically computed colours, and includes a very neat demo from @cdruan
A thread discussing using the colour macro as an attribute value - https://talk.tiddlywiki.org/t/how-to-use-the-colour-macro-as-an-attribute-value/10051
A thread discussing improvements to colour handling, including a similar idea for palette previews - https://talk.tiddlywiki.org/t/idea-approach-to-palettes-and-themes/9431

Progress

Documentation
- Palettes and colour schemes
- Media Query Tracker
- Background actions
- changecount operator
- colour-interpolate operator
- colour-lighten operator
- colour-darken operator
- colour-get-oklch operator
- colour-set-oklch operator
- colour-contrast operator
- colour-best-contrast operator

@linonetwo

The replacement library from https://colorjs.io/ is much, much larger but I think we can develop a custom build that uses treeshaking to whittle the code down to the bits that we need. @linonetwo does that sound feasible? I intend the explore further improvements but I wanted to start by establishing a library that can do modern P3 and OKLCH colour calculations.

Really just syntactic sugar for the wikify widget

Using the new wikify operator. Currently has a bug whereby redirected colours (like "tiddler-background") do not work. Direct colours like "background" do work. Note the hacks needed to makeFakeWidgetWithVariables work

github-actions · 2024-10-25T09:02:22Z

Confirmed: Jermolene has already signed the Contributor License Agreement (see contributing.md)

themes/tiddlywiki/vanilla/base.tid

core/wiki/paletteswitcher.tid

core/modules/utils/dom/color.js/tiddlywiki.files

kookma · 2024-11-04T03:38:02Z

Is there a demo to test colour handling improvement?

Jermolene · 2024-11-04T18:18:08Z

Is there a demo to test colour handling improvement?

Hi @kookma I've posted a preview to tiddlyhost

See #8702 (comment)

Jermolene · 2025-02-12T20:31:14Z

Very promising developments. The "ink" and "paper" metaphors are intuitive, and interpolation effects seem to be working well!

Thanks @springerspandrel

Sidebar tabs are not yet "inheriting" the tab-background and tab-background-selected palette values in TwentyTwenties (though it seems that's the intention of having these default to [function[colour],[tab-background]] (etc.)

That should be fixed now.

(a) tab-background and tab-background-selected colors don't get any preview even though sidebar tabs are very much part of "out-of-box" GUI... Having a mock set of sidebar tab-rectangles (one active, one or two inactive) might be helpful, at the risk of more visual noise... (Meanwhile, we do currently get a preview of search window input-box — despite duplicating tiddler background for all bundled palettes except Solarized Dark.)

The previews now show sidebar tabs in the appropriate colours.

(b) Selecting new primary color doesn't affect palette's preview-thumbnail at all.

That should be fixed now.

(ONE solution: Include minimalist mock-preview of sidebar tab list content — so the tiddler-link color appears in preview, whether or not a palette has distinct value for site-title-foreground ... ALTERNATE solution: have some primary link-color "accents" within the top mock tiddler such as HelloThere — noting that the real HelloThere conveniently does feature several visible links within a short tiddler body!).

Both suggestions are now implemented.

(c) Selecting new secondary and/or tertiary color choices doesn't affect palette preview

That should be fixed now.

and user also lacks obvious places — in larger interface — to recognize dynamic effects of that selection. If the second "mock" tiddler in preview-thumbnail were to include a small block of code, that would nicely gives a hint of tertiary.

A block of code is an excellent idea.

(Including a natural preview of the secondary color seems more difficult, since having an alert within preview thumbnail is awkward.)

I have included an alert in the latest update, but the challenge is that it obscures the title of the first tiddler. Perhaps previews need multiple pages to show things like alerts, notifications and modals.

pmario · 2025-02-12T20:42:00Z

core/ui/PaletteEditor.tid

@@ -0,0 +1,94 @@
+title: $:/PaletteEditor


Indentation please.

Just to be clear, this file hasn't been changed (yet), it is just moved from $:/PaletteManager

I meant to add that this file needs substantial revision so I will add indentation when I do that.

pmario · 2025-02-14T11:46:09Z

core/wiki/macros/CSS.tid

@@ -3,8 +3,9 @@ tags: $:/tags/Macro



tags: $:/tags/Global is missing

Hi @pmario all the core macros use $:/tags/Macro rather than $:/tags/Global. I think we kept it that way to be super cautious about backwards compatibility. We can discuss changing them, but they should all be changed at the same time.

IMO we should use $:/tags/Global for newly introduced core macros,widgets and functions, while $:/tags/Macro for old core macros.

IMO we should use $:/tags/Global for newly introduced core macros,widgets and functions, while $:/tags/Macro for old core macros.

That's reasonable. $:/core/macros/CSS is of course old, but I may do further refactoring of this PR.

pmario · 2025-02-14T11:52:49Z

@Jermolene .. Am I right, that palettes are becoming a "description" how they should be "consolidated" and that they can be chained.

Is this PR ready to be tested or should I wait?

Jermolene · 2025-02-14T13:28:38Z

@Jermolene .. Am I right, that palettes are becoming a "description" how they should be "consolidated" and that they can be chained.

I wouldn't think of it that way.

The consolidation process is just part of the implementation of the palette import process. It's not strictly necessary but it's helpful for debugging to be able to see the final values that were used to compose the palette.

Is this PR ready to be tested or should I wait?

Do you mean should you click through to the demo? It's probably hard to assess this PR without trying out the demo. It is still a draft, and there's more to do.

pmario · 2025-02-14T11:58:26Z

core/modules/background-actions.js

+
+BackgroundActionDispatcher.prototype.trackFilter = function(title) {
+	var tiddler = this.wiki.getTiddler(title),
+		id = this.nextTrackedFilterId++,


I think this counter this.nextTrackedFilterId should be named this.nextTrackedFilterIdx ... Idx for index or Cnt for counter. Just a thought. I did not single step the function yet

Thanks @pmario. These ids are intended to be opaque to client code; there is no semantics in incrementing or decrementing them, they're just used as an identifier.

pmario · 2025-02-14T12:07:24Z

core/modules/background-actions.js

+	this.wiki = options.wiki;
+	this.title = options.title;
+	this.trackFilter = options.trackFilter;
+	this.actions = options.actions


semicolon missing at line end?

Thanks @pmario

pmario · 2025-02-14T12:58:14Z

core/wiki/macros/tag.tid

@@ -28,15 +28,15 @@ color:$(foregroundColor)$;
 </$let>
 \end

-\define tag-pill-body(tag,icon,colour,palette,element-tag,element-attributes,actions)
+\define tag-pill-body(tag,icon,colour,element-tag,element-attributes,actions)


hmm, IMO we can not change this macro signature because of backwards compatibility reasons.

In the docs we need to deprecate the parameter.

Thanks @pmario

Rhys-T · 2025-02-14T14:30:31Z

core/wiki/paletteswitcher.tid

+
+<$linkcatcher actions=<<actions>>>
+	<div class={{{ tc-chooser [<thumbnails>match[yes]then[tc-chooser-cards]] +[join[ ]] }}}>
+		<$list filter="[all[shadows+tiddlers]tag[$:/tags/Palette]sort[name]]">


One minor thing I noticed is that the palette switcher shows any draft versions of palettes, in addition to the actual saved palettes. (This is actually present in the current version too, but there it could plausibly be almost useful if you were trying to edit a palette with the regular tiddler editor and see the changes live. That doesn't work with the new compilation system, though.)

Thanks @Rhys-T I fixed this in master in 3051e8d

pmario · 2025-02-14T16:19:35Z

I did the following tests using Windows 11 pro, FireFox 135

Selecting AutoToggle with default settings
Change Windows dark / light -> Works
Overwrite Setting in FireFox -> Works
Change dark / light in FireFox -> Works

Import my palette-switcher plugin -> Switching works, but it messes up the PR $:/PaletteManager
May be we need a second $:/PaletteManagerNew -- TODO I will need a bit more testing.

See screenshot

Import my palette-watch plugin, which needs a save and reload -> Works
AutoToggle -> Works
My Plugins -> Work
FireFox setting uses my plugin -> Works
Windows colour settings use my plugin -> Works

Jermolene · 2025-02-14T16:46:43Z

Thanks for trying things out @pmario.

Import my palette-switcher plugin -> Switching works, but it messes up the PR $:/PaletteManager

Weird. The preview is showing thumbnails, but $:/snippets/paletteswitcher is written to default to omitting the thumbnails, and requires an explicit parameter to do so.

See screenshot

The fact that the styling in $:/PaletteManager has been changed by the palette-switcher plugin suggests that that plugin is redefining core styles, which would usually be considered bad practice.

These are mostly RGB entries that were previously missing, filled in with values from Vanilla. The goal is still not to have any direct RGB colours in the palette, just computed colours derived from the base colours

Makes debugging easier, and works in CSS and as a style.prop assignment

pmario · 2025-03-07T16:30:32Z

core/palettes/TwentyTwenties.tid

-button-border:
+button-background: 
+button-border: 
+button-foreground: 


Do we need spaces here?

I intend to fill in those entries later.

pmario · 2025-03-07T16:35:39Z

core/palettes/TwentyTwenties.tid

@@ -76,7 +78,7 @@ dropdown-background: [tf.colour[background]]
 dropdown-border: [tf.colour[muted-foreground]]
 dropdown-tab-background-selected: [tf.colour[background]]
 dropdown-tab-background: [tf.interpolate-colours[base-paper],[base-ink],[0.9]]
-dropzone-background: rgba(0,200,0,0.7)
+dropzone-background: [tf.colour[base-secondary]colour-set-alpha[0.7]]


I do like this possibility: [tf.colour[base-secondary]colour-set-alpha[0.7]] but once the "alpha" values are settled, would it be possible to have eg: colour-set-alpha<lighter>, colour-set-alpha<lighter> colour-set-alpha<lightest> and eg: dark, darker, darkest -- What do you think?

In the same realm. I know colour-set-alpha is the right technical term, but our users would probably prefer darken-colour<percent>

Question Does "alpha" set transparency or does it calculate a lighter or darker colour?

Hi @pmario to darken a colour one would use colour-set-oklch:l[0.5] to set the lightness.

We don't yet have operators to extract the components of a colour in a particular colour space, but once we have those then it would be possible to make macros for setting the alpha or brightness in relative steps.

Question Does "alpha" set transparency or does it calculate a lighter or darker colour?

Apologies, to be clear it sets transparency

Jermolene added 11 commits October 21, 2024 09:03

Include colour.js license and mark version number

93d1c05

Introduce wikify operator

467a1a4

Really just syntactic sugar for the wikify widget

Rewrite colour macro as a function

a23ee16

Using the new wikify operator. Currently has a bug whereby redirected colours (like "tiddler-background") do not work. Direct colours like "background" do work. Note the hacks needed to makeFakeWidgetWithVariables work

Merge branch 'master' into colour-improvements

e1e73d2

Fix typo

1e5c69e

Fix nested colour definitions

4af573a

Fix up the tests

96b85ed

Add colour-lighten and colour-darken operators

d372729

Remove obsolete comment

b5a22e3

Improve colour palette switcher with previews

fc36941

Palette chooser styling tweaks

151f61a

pmario reviewed Oct 25, 2024

View reviewed changes

themes/tiddlywiki/vanilla/base.tid Show resolved Hide resolved

Jermolene added 5 commits October 30, 2024 20:38

Merge branch 'master' into colour-improvements

4445111

Merge branch 'master' into colour-improvements

4f2754d

Fix palette chooser when displayed in "Tools" dropdown

bc0fde6

Merge branch 'master' into colour-improvements

85fa913

Add expertimental colour-oklch operator

a8fb071

pmario reviewed Nov 3, 2024

View reviewed changes

core/wiki/paletteswitcher.tid Show resolved Hide resolved

pmario reviewed Nov 3, 2024

View reviewed changes

core/modules/utils/dom/color.js/tiddlywiki.files Show resolved Hide resolved

Jermolene added 2 commits November 4, 2024 10:39

Merge branch 'master' into colour-improvements

81b7bb4

Indentation for palette switcher

ff5c846

Jermolene added 5 commits November 4, 2024 19:26

Merge branch 'master' into colour-improvements

23eccd1

Restore default styling for chosen chooser item

c607440

See #8702 (comment)

Remove logging

348f717

Testing Netlify CI

c3ce9ca

Testing CI

a4d9303

Add temporary guide tiddler at the top

377856c

pmario reviewed Feb 12, 2025

View reviewed changes

Jermolene added 2 commits February 13, 2025 09:10

Background action demos should require explicit enabling

0037813

Introduce tf.colour function to make palette entries more concise

d7df7ed

pmario reviewed Feb 14, 2025

View reviewed changes

Merge branch 'master' into colour-improvements

8b05b72

pmario reviewed Feb 14, 2025

View reviewed changes

Fixes suggested by @pmario

2028420

Rhys-T reviewed Feb 14, 2025

View reviewed changes

Merge branch 'master' into colour-improvements

059e439

Jermolene mentioned this pull request Feb 18, 2025

Introducing bundled sub plugins #8957

Draft

10 tasks

Jermolene added 2 commits February 18, 2025 16:54

Merge branch 'master' into colour-improvements

93f9544

Merge branch 'master' into colour-improvements

75ba085

saqimtiaz mentioned this pull request Feb 28, 2025

[IDEA] SetTimeout / SetInterval / SetAlarmClock widget #8967

Open

Jermolene added 9 commits March 6, 2025 21:38

Merge branch 'master' into colour-improvements

8467fa3

Fix from mismerging from master

624cf95

TwentyTwenties: copy over missing palette entries

250ee90

These are mostly RGB entries that were previously missing, filled in with values from Vanilla. The goal is still not to have any direct RGB colours in the palette, just computed colours derived from the base colours

Preview: Fix sidebar link colour

3001696

Include palette entry name in palette entries

ee977de

Makes debugging easier, and works in CSS and as a style.prop assignment

Add colour-set-alpha operator

d42e3d3

Fix colour-interpolate with missing colours

4c95ae5

TwentyTwenties improvements

5e27462

Fix tests

5897b82

pmario reviewed Mar 7, 2025

View reviewed changes

Jermolene mentioned this pull request Mar 7, 2025

Introduce let filter run prefix for assigning a filter run result to a variable #8972

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Colour handling improvements #8702

Colour handling improvements #8702

Jermolene commented Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 25, 2024

kookma commented Nov 4, 2024

Jermolene commented Nov 4, 2024

Jermolene commented Feb 12, 2025

pmario Feb 12, 2025

Jermolene Feb 12, 2025

pmario Feb 12, 2025

Jermolene Feb 12, 2025

pmario Feb 14, 2025

Jermolene Feb 14, 2025 •

edited

Loading

Leilei332 Feb 14, 2025

Jermolene Feb 14, 2025

pmario commented Feb 14, 2025

Jermolene commented Feb 14, 2025

pmario Feb 14, 2025

Jermolene Feb 14, 2025

pmario Feb 14, 2025

pmario Feb 14, 2025

Jermolene Feb 14, 2025

pmario Feb 14, 2025

Jermolene Feb 14, 2025

Rhys-T Feb 14, 2025

Jermolene Feb 14, 2025

pmario commented Feb 14, 2025 •

edited

Loading

Jermolene commented Feb 14, 2025

pmario Mar 7, 2025

Jermolene Mar 7, 2025

pmario Mar 7, 2025

pmario Mar 7, 2025

Jermolene Mar 7, 2025

Jermolene Mar 7, 2025

Colour handling improvements #8702

Are you sure you want to change the base?

Colour handling improvements #8702

Conversation

Jermolene commented Oct 25, 2024 • edited Loading

Introduction

Try the Demo

Changes to Palettes

Palette Compilation

Automatic Palette Readability Tests

Colour Manipulation

Media Query Tracker

Background Actions

apply Filter Run Prefix

Backwards Compatibility

References

Progress

github-actions bot commented Oct 25, 2024

kookma commented Nov 4, 2024

Jermolene commented Nov 4, 2024

Jermolene commented Feb 12, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jermolene Feb 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmario commented Feb 14, 2025

Jermolene commented Feb 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmario commented Feb 14, 2025 • edited Loading

Jermolene commented Feb 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jermolene commented Oct 25, 2024 •

edited

Loading

`apply` Filter Run Prefix

Jermolene Feb 14, 2025 •

edited

Loading

pmario commented Feb 14, 2025 •

edited

Loading