Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

StackStorm v3.2dev pre-release Testing #6

Closed
arm4b opened this issue Apr 17, 2020 · 23 comments
Closed

StackStorm v3.2dev pre-release Testing #6

arm4b opened this issue Apr 17, 2020 · 23 comments
Labels
community release
Milestone
3.2.0

Comments

@arm4b
Copy link
Member

arm4b commented Apr 17, 2020 •
edited
Loading

We're ready to prepare the StackStorm v3.2 release and starting pre-release testing.

This will be the first StackStorm release under the Linux Foundation with new Open Source Governance and Maintainers team. We now rely on our Community more than ever and invite everyone to help with the pre-release testing and development/contribution.

Release Process Preparation

Per Release Management Schedule @armab is the Release Manager and @punkrokk Assisting for v3.2. They will freeze the master for the major repositories in StackStorm org, follow the StackStorm Release Process which is now available to public, accompanied by the Useful Info for Release managers. Communication is happening in #releasemgmt and #development Slack channels.
The first step is pre-release manual user-acceptance testing for v3.2dev.

Why Manual testing?

StackStorm is very serious about testing and has a lot of it: Unit tests, Integration, Deployment/Integrity checks, Smoke tests and eventually end-2-end tests when automation spins up new AWS instance for each OS/flavor we support, installs real st2 like user would and runs set of st2tests (for each st2 PR, nightly, periodically, during release).

See st2ci and st2cd for more examples and workflows about how StackStorm automation is used to test StackStorm (dogfooding).

That's a perfect way to verify what we already know and codify expectations about how StackStorm should function.

However it's not enough.
There are always new unknowns to discover, edge cases to experience and tests to add. Hence, manual Exploratory Testing is an exercise where entire team gathers together and starts trying (or breaking) new features before the new release. Because we're all different, perceive software differently and try different things we might find new bugs, improper design, oversights, edge cases and more.

This is how StackStorm previously managed to land less major/critical bugs into production.

TL;DR

Install StackStorm v3.2dev staging packages, try random things in random environments (different OS) and report any regressions found comparing to v3.1:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=Ch@ngeMe --unstable

Extra points for PR hotfixes and adding new or missing test cases.

Major changes

Important changes which are highly encouraged to try and test.

Full Changelog

Changes which are recommended to ack, explore, check and try in a random way.

st2

Added

Changed

  • Install pack with the latest tag version if it exists when branch is not specialized.
    (improvement) Install pack with the latest tag st2#4743

  • Implement "continue" engine command to orquesta workflow. (improvement) Orquesta workflows have incorrect status st2#4740

  • Update various internal dependencies to latest stable versions (apscheduler, eventlet,
    kombu, amqp, pyyaml, mongoengine, python-gnupg, paramiko, tooz, webob, bcrypt).

    Latest version of mongoengine should show some performance improvements (5-20%) when
    writing very large executions (executions with large results) to the database. Update various Python dependencies to latest stable versions st2#4767

  • Improved development instructions in requirements.txt and dist_utils.py comment headers
    (improvement) Fix dependency conflicts st2#4774

  • Add new actionrunner.stream_output_buffer_size config option and default it to -1
    (previously default value was 0). This should result in a better performance and smaller
    CPU utilization for Python runner actions which produce a lot of output.
    (improvement)

    Reported and contributed by Joshua Meyer (@jdmeyer3) fixing subprocess to use system buffer instead of being unbuffered st2#4803

  • Add new action_runner.pip_opts st2.conf config option which allows user to specify a list
    of command line option which are passed to pip install command when installing pack
    dependencies into a pack specific virtual environment. Don't call eventlet directly inside the st2reactor code st2#4792

  • Refactor how orquesta handles individual item result for with items task. Before the fix,
    when there are a lot of items and/or result size for each item is huge, there is a negative
    performance impact on write to the database when recording the conductor state. (improvement)

  • Remove automatic rendering of workflow output when updating task state for orquesta workflows.
    This caused workflow output to render incorrectly in certain use case. The render_workflow_output
    function must be called separately. (improvement)

  • Update various internal dependencies to latest stable versions (cryptography, jinja2, requests,
    apscheduler, eventlet, amqp, kombu, semver, six) Update pip dependencies, fix conflicting dependency version, ensure there are no conflicting dependencies st2#4819 (improvement)

  • Improve MongoDB connection timeout related code. Connection and server selection timeout is now
    set to 3 seconds. Previously a default value of 30 seconds was used which means that for many
    connection related errors, our code would first wait for this timeout to be reached (30 seconds)
    before returning error to the end user. Fix SSL support for MongoDB and RabbitMQ under Python 3.x st2#4834

  • Upgrade pymongo to the latest stable version (3.10.0.). Update pymongo to the latest version st2#4835 (improvement)

  • Updated Paramiko to v2.7.1 to support new PEM ECDSA key formats Fixes: #4890 to support new ssh key types st2#4901 (improvement)

  • Remove .scrutinizer.yml config file. No longer used.

  • Convert escaped dict and dynamic fields in workflow db models to normal dict and dynamic fields.
    (performnce improvement)

  • Add support for PEP 508 <https://www.python.org/dev/peps/pep-0508/stackstorm/st2#environment-markers>_
    environment markers in generated requirements.txt files. (improvement) Various requirements fixups st2#4895

  • Use pip-compile from pip-tools instead of pip-conflict-checker (improvement) Fix futurist version conflict st2#4896

  • Refactor how inbound criteria for join task in orquesta workflow is evaluated to count by
    task completion instead of task transition. (improvement)

  • The workflow engine orquesta is updated to v1.1.0 for the st2 v3.2 release. The version upgrade
    contains various new features and bug fixes. Please review the release notes for the full list of
    changes at https://github.com/StackStorm/orquesta/releases/tag/v1.1.0 and the st2 upgrade notes
    for potential impact. (improvement)

Fixed

Removed

Full list of changes: https://github.com/StackStorm/st2/blob/master/CHANGELOG.rst

orquesta

Added

  • Add feature to retry task execution on given condition with delay in between retries and
    max number of retries. (new feature)
    Contributed by Hiroyasu Ohyama (@userlocalhost) and Nick Maludy (@nmaludy)
  • Add support to rerun completed workflow (failed or succeeded) from given task(s). (new feature)

Changed

  • Replace "noop" with "continue" when "do" is not specified. The new "continue" command
    will not alter the previous task state and will continue to conduct the workflow
    execution. Orquesta workflows have incorrect status st2#4740 (improvement)
  • Refactor conductor to not store each item result in task state. If there are a lot of items
    and/or result size is huge per item, then there will be a performance impact on database
    write operations when recording the conductor state. (improvement)
  • Use ujson to deepcopy dict(s) for faster performance. (improvement)
  • Refactor how inbound criteria for join task is evaluated to count by task completion
    instead of task transition. (improvement)

Fixed

Fore more info see https://github.com/StackStorm/orquesta/blob/master/CHANGELOG.rst

st2chatops

st2web

  • No changes.

Warning!
st2web UI needs a contributor/maintainer! Security alerts due to outdated dependencies, as well as tech debt, issues and number of unreviewed PRs is all high.

This could be huge. With almost 1 year since v3.1 st2 release codebase accumulated so many new features, changes and fixes.

Please report findings here and bugs/regressions in respective repositories.
Depending on severity and importance bugs might be fixed before the release or postponed to the next release if they're very minor and not a release blocker.
@arm4b arm4b added help wanted Extra attention is needed community release labels Apr 17, 2020
@arm4b arm4b added this to the 3.2.0 milestone Apr 17, 2020
@arm4b
Copy link
Member Author

arm4b commented Apr 17, 2020

@StackStorm/maintainers Even if you didn't find any bugs, please report back in this discussion which OS/environment you've used and which features/items you've tried or verified. This will help to get an overall idea about OS/distro testing distribution.
I may assign OS/flavors to different team members as we did before.

@amanda11
Copy link

Very brief test done on centos8. Installed using single line install which went fine.

Simple chatops configured with slack for the aws ec2 get alias (note: stackstorm_aws 1.2.3 pack used, as 1.3.1 had issues - which I'll report separately - but not to do with stackstorm version).
Simple orquesta workflow moved over successfully from stackstorm 3.1 install, which used the aws_boto3 and slack packs.

I did fall foul to the change where it installs the latest tagged pack rather than latest version of the code, on page: https://docs.stackstorm.com/latest/packs.html it states that it installs the latest commit of a pack rather than the tagged, but looking at the change log then that seems to be a known change.
Docs state "By default, the latest commit to a pack repository will be installed, but you can specify a particular version, branch, tag, or even a commit hash. Just use =:" yet, it seems to install last tag. If this is by design as per "Install pack with the latest tag version if it exists when branch is not specialized. (improvement) #4743" - then does documentation need updating?

@arm4b
Copy link
Member Author

arm4b commented Apr 21, 2020

@amanda11 Awesome, thank you very much for testing and feedback!

Yes indeed, in 3.2dev the behavior of pack install has changed which now installs latest "tag/version" instead of latest commit from master.
That's a good catch! 👍

@amanda11 Documentation indeed needs updating. How about submitting a small PR to fix that?
https://github.com/StackStorm/st2docs/blob/master/docs/source/packs.rst

@amanda11
Copy link

@armab I'll submit a PR for the doc change. thanks.

@amanda11
Copy link

@armab I'll submit a PR for the doc change. thanks.

StackStorm/st2docs#973 raised. I also verified the behaviour when you had a local repo that it was working, and what it picked up if you used file:/// towards a local git repo. That seems to still use the latest commit when using a file:/// to a local repo - so I left that section stating latest commit. So I literally just changed one word in the docs in the end.

@arm4b arm4b mentioned this issue Apr 21, 2020
Merged
@arm4b
Copy link
Member Author

arm4b commented Apr 21, 2020

FYI for someone testing 3.1 -> 3.2 upgrade scenarios here is how to install 3.1dev from unstable packages first:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- \
  --unstable --version=3.1dev \
  --user=st2admin --password=Ch@ngeMe

Then follow the packages upgrade procedure https://docs.stackstorm.com/latest/install/upgrades.html#general-upgrade-procedure to get v3.1dev->v3.2dev upgraded.

@arm4b
Copy link
Member Author

arm4b commented Apr 21, 2020 •
edited
Loading

Installation with the following instructions fails on EL6 and EL7:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s --   --unstable --version=3.1dev   --user=st2admin --password=Ch@ngeMe

repoquery -y flag which was added before is non-existent for EL6.
Update: -y is non-existent for EL7 as well.

Reported as StackStorm/st2-packages#641 (Fixed now)

@arm4b arm4b mentioned this issue Apr 21, 2020
Merged
@amanda11
Copy link

Further positive feedback on testing. Loving the immutable parameters on the aliases. Needed them for some aliases I needed for AWS reporting using the aws_boto3 pack. Integrated those successfully with st2chatops using 3.2dev on Centos8.

This was referenced Apr 23, 2020
Closed
Merged
@arm4b
Copy link
Member Author

arm4b commented Apr 23, 2020 •
edited
Loading

Another 2 minor issues/enhancements were spot:

Both are not release blockers and will be planned for the next st2 v3.3.

@amanda11
Copy link

Probably not an issue - but a problem I encountered. If I already have a config for a pack under /opt/stackstorm/configs, and then try and use st2 pack config then I get an error due to failing to have write permission to that file. It works if I delete the old config file, as it then creates a new file with the owner of st2. But I am guessing that if people have manually configured files they may not have created them with the owner as st2.

@amanda11
Copy link

I had a play with the retry feature in the Orquesta workflow and I was surprised that I didn't have to up the version in the workflow yaml to state it was using version 1.1.0. I have the version as 1.0 and it works fine (documentation example uses version 1.0).
So although new functionality it is still the same version of the workflow DSL?

@amanda11
Copy link

amanda11 commented Apr 23, 2020 •
edited
Loading

I have a Orquesta workflow with:
action A - with retry count of 2 (no when clause on the retry - so default to failed)
action B - slack post message
The when transition on A goes to B if A fails.
I was expecting to only goto B once. But I'm seeing that B is being called for all 3 failures. Whereas I expected that it would only do the fail transition once.

So when the first attempt at action A fails, it does two things:

  1. trigger action A again
  2. goes to the action I have configured to run when A fails

Thus resulting in the failure message being output 3 times rather than as I would have expected it to just get the failure after all attempts had failed.

@arm4b
Copy link
Member Author

arm4b commented Apr 23, 2020

@amanda11 For the orquesta with retry, that sounds unexpected to me too.
From the description, could be somewhat related to StackStorm/orquesta#138 (comment)

Can you please create a new issue in https://github.com/StackStorm/orquesta/ with example workflow for @m4dcoder to reproduce easily, describing what's expected and what went wrong?

@m4dcoder
Copy link

I had a play with the retry feature in the Orquesta workflow and I was surprised that I didn't have to up the version in the workflow yaml to state it was using version 1.1.0. I have the version as 1.0 and it works fine (documentation example uses version 1.0).
So although new functionality it is still the same version of the workflow DSL?

The version in WF DSL is a placeholder currently and the engine don't support multiple DSL versions. The new features are additive. In the future as we make major changes such as schema change or deprecation, we will definitely need to figure out how to separate support of different DSL versions.

@m4dcoder
Copy link

Here's the fix for the task retry reported above. PR @ StackStorm/orquesta#200. Good catch @amanda11, keep them coming. @armab Once PR is merged, I'm thinking of republishing the v1.1.0 release for orquesta. What do you think?

@winem
Copy link

winem commented Apr 24, 2020

Hi,

here is the summary of my tests from tonight:

  • installation and basic st2 commands on CentOS8
  • installation and basic st2 commands on Ubuntu 18.04 (all tests mentioned below were made on this system)
  • tested the automatic installation of dependencies (#4769)
    • install local pack with dependencies
    • install local pack with dependency to another local pack with dependencies to github & st2 exchange
    • install several packs with the same dependencies
    • install pack with dependencies to local and remote packs
    • install a pack (pack1) with a dependency to pack2 which has a dependency to pack1 (to see if this creates some kind of infinite loop or deadlock)
  • test the new continue command (#4740) and the retry feature in Orquesta workflows
  • compare the performance of with item tasks on st2 3.1 vs. 3.2 (Fix schema of with items task to not allow additional properties orquesta#195)
  • test a bunch of actions from the aws_boto3 pack and verify that existing workflows still work
  • tested inquiries with properties of all available types
  • tested the pack API
    • only the tests with the tags filter failed (see #4918)

I hope this helps. Will continue testing in the next days. Let me know if you want anything special tested. Otherwise I will just keep going through the release notes and test the mentioned changes.

@amanda11
Copy link

amanda11 commented Apr 24, 2020 •
edited
Loading

Whilst testing re-run from task functionality I raised: StackStorm/st2docs#975 on the documentation. Have submitted PR to just remove those that are included, but may want to add new features to the list if any are known.

@arm4b
Copy link
Member Author

arm4b commented Apr 24, 2020 •
edited
Loading

@m4dcoder For the Orquesta hotfix I think it's best to publish new v1.1.1 patch to include in st2.
Also replied in StackStorm/orquesta#200 (comment)

@arm4b
Copy link
Member Author

arm4b commented Apr 25, 2020

Current 3.2dev pre-release testing status:

@amanda11
Copy link

amanda11 commented Apr 27, 2020 •
edited
Loading

Documentation issue raised on upgrade docs when running on centos 8 StackStorm/st2docs#977.
Proposed st2docs PR to split rhel8/centos 8 so that skip the mistral packages: StackStorm/st2docs#978

@arm4b
Copy link
Member Author

arm4b commented Apr 28, 2020

Next stage of release automation process & testing, following the https://github.com/StackStorm/discussions/wiki/Release-Process#process-for-major-release (now open) is that we got staging-stable packages out in https://packagecloud.io/stackstorm/staging-stable repository.

At this stage we can just quickly check the packages like integrity/smoke tests to make sure this version is not different to what we tested last week and has no surprises:

curl -sSL https://stackstorm.com/packages/install.sh | bash -s -- --user=st2admin --password=st2admin --staging

These packages are now candidate for the final stable promotion.
If no major issues found, we'll continue the release automation tomorrow, 28 Apr, US morning.

FYI for someone wondering the pipeline is the following:
staging-unstable -> unstable -> staging-stable -> stable.
See https://github.com/StackStorm/discussions/wiki/Useful-info-for-Release-Managers#packagecloud-repositories for more info about each repository.

@winem
Copy link

winem commented May 1, 2020

Probably a bit late to the party but I just ran all the tests from my previous comment on the staging packages again - successfully.

@arm4b
Copy link
Member Author

arm4b commented May 1, 2020 •
edited
Loading

Thanks everyone!
Especially @amanda11 and @winem for extra effort catching and fixing bugs, - that was very helpful.

We now have StackStorm v3.2.0 stable out:
https://stackstorm.com/2020/04/30/stackstorm-v3-2-0-released/

@arm4b arm4b removed the help wanted Extra attention is needed label May 1, 2020
This was referenced May 8, 2020
Merged
Open
@arm4b arm4b mentioned this issue Jun 22, 2020
Merged
@blag blag mentioned this issue Nov 17, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community release
Projects
None yet
Milestone
3.2.0
Development

No branches or pull requests
5 participants
@winem @arm4b @cognifloyd @m4dcoder @amanda11