Cannot add_route in PHASE0_CONFIG if using view derivers

[dstufft]

From what I can tell, if you attempt to use view derivers you cannot call Configurator().add_route() inside of a action where order=PHASE0_CONFIG. Using the following code to reproduce:

from pyramid.config import PHASE0_CONFIG, Configurator
from pyramid.response import Response


def no_op_view(view, info):
    return view

no_op_view.options = {"noop"}


def add_a_thing(config, thing, *args, **kwargs):
    def register():
        config.add_route(thing, *args, **kwargs)
    config.action(("thing", thing), register, order=PHASE0_CONFIG)


def hello_world(request):
    return Response('Hello %(name)s!' % request.matchdict)


config = Configurator()
config.add_view_deriver(no_op_view)
config.add_directive("add_a_thing", add_a_thing)
config.add_a_thing("my-thing", "/")
config.add_route('hello', '/hello/{name}')
config.add_view(hello_world, route_name="hello", noop=True)
config.commit()

If you execute that is, you'll get an error like:

Traceback (most recent call last):
  File "demo2.py", line 27, in <module>
    config.commit()
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 654, in commit
    self.action_state.execute_actions(introspector=self.introspector)
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 1158, in execute_actions
    list(pending_actions) +
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 1121, in resume
    for a, b in zip_longest(actions, executed_actions):
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/__init__.py", line 1255, in resolveConflicts
    discriminator = undefer(action['discriminator'])
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/registry.py", line 271, in undefer
    v = v.resolve()
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/registry.py", line 263, in resolve
    return self.value
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/decorator.py", line 45, in __get__
    val = self.wrapped(inst)
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/registry.py", line 260, in value
    return self.func()
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/views.py", line 810, in discrim_func
    self._check_view_options(**dvals)
  File "/Users/dstufft/.virtualenvs/tmp-afd03b5a92f8d7f/lib/python3.5/site-packages/pyramid/config/views.py", line 1070, in _check_view_options
    raise ConfigurationError('Unknown view options: %s' % (kw,))
pyramid.exceptions.ConfigurationError: Unknown view options: {'noop': True}

However, if you do one of:

• Remove the call to config.add_route inside of the action.
• Remove the noop=True parameter to config.add_view().
• Change the order=PHASE0_CONFIG to order=PHASE1_CONFIG.

Then the code will execute successfully, although I assume the third option will break if you attempt to call something that is executed in Phase 1 instead of add_route which is executed in Phase 2.

[mmerickel]

After a little sleuthing I've determined that the deferred discriminators are being undeferred at the start of the conflict resolution sort in resolveConflicts. This is an issue specific to the add_view discriminator which touches predicates and view options. It's the only action using a deferred discriminator.

I'm looking into how to fix this, but wanted to leave some notes here before going to bed.

[mmerickel]

Ah, it occurs on a re-entrant sort (triggered by the add_route inside of register). resolveConflicts sorts all the actions it receives, without regard for where we're currently at in the order. It needs to stop sorting until the execution has caught up in order.

[mmerickel]

resolveConflicts is currently written with two important, and not very clear, assumptions. 1) It can test discriminators whenever it wants (this is breaking in the case of the deferred discriminator) and 2) conflict resolution only occurs between actions in the same order (you cannot use a discriminator in order A to affect an action/discriminator in order B).

I do not see a way to preserve both of these assumptions and since assumption 2 has been in place longer, I'll err toward keeping that one. It's not possible to do conflict resolution across orders AND leave some of the discriminators in a deferred state. It just doesn't work.

If we break assumption 1 then it means we should only undefer / sort actions in the order we are executing and never undefer / sort actions that come in a later order. This should be possible but will require somewhat of a rewrite of the execute_actions and resolveConflicts logic.

[mmerickel]

Crap, now I've dug too far into this. There were some traces of code in resolveConflicts that didn't really make sense regarding the order and so I looked further. The conflict resolution algorithm actually changed in undocumented ways when the deferred discriminator was introduced. Prior to pyramid 1.4, discriminators worked across orders.

9c8ec5c

[mmerickel]

Spent a little while longer thinking about this - it's something I ran into while implementing the re-entrant stuff originally but missed this aspect. There are basically three options.

1. Figure out how to get rid of the deferred discriminator.
2. Document and live with the fact that discriminators can only affect actions of the same order.
3. Figure out how to resolve conflicts on a deferred discriminator.

Unfortunately, I think option 2 is the only option as I cannot see a way to get rid of the deferred discriminator. It's solving a real problem where the discriminator itself cannot be computed until actions from an earlier order have executed.

The only solution to option 3 that I can think of is to execute action A and then later when we undefer the discriminator for action B, if it ends up conflicting with the one from action A such that it has a higher priority then we have no choice but to error.

The only solution to option 3 that I can think of is to execute action A and then later when we undefer the discriminator for action B, if it ends up conflicting with the one from action A such that it has a higher priority then we have no choice but to error. However if we stick with an approach like option 2 then we can just allow the author to define 2 actions with the same discriminator on different orders to kill action A(order=0) and allow action B(order=1) to execute. The only downside is that the author needs to know, in advance, for each action he wants to override, what the order is of the action.

[mmerickel]

@dstufft this should be fixed in #2757. I am not yet sure whether I should release this as a bug fix or hold off until 1.8 as its yet another rewrite of pyramid's config execution pipeline. I'm open to opinions!