Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking Issue: Make Harnesses Optional #3832

Open
3 of 12 tasks
carolynzech opened this issue Jan 14, 2025 · 2 comments
Open
3 of 12 tasks

Tracking Issue: Make Harnesses Optional #3832

carolynzech opened this issue Jan 14, 2025 · 2 comments
Assignees
Labels
[C] Feature / Enhancement A new feature request or enhancement to an existing feature.

Comments

@carolynzech
Copy link
Contributor

carolynzech commented Jan 14, 2025

Requested feature: Make proof harnesses optional, when possible
Use case: There are plenty of "boilerplate harnesses" that look something like this:

fn foo(x: u8, y: u8) { ... }

#[kani::proof]
fn foo_proof() {
  let x = kani::any();
  let y = kani::any();
  foo(x, y);
}

There's nothing surprising in this harness--the user just creates arbitrary variables for each argument of the function they're verifying, and then calls the function. Ideally, the user shouldn't have to write this harness at all; Kani should detect that each of the arguments implements Arbitrary, then attempt to verify it automatically.

Tasks

(We define "standard harness" as a #[kani::proof] harness and a "contract harness" as a #[kani::proof_for_contract] harness).
I'll break these tasks down into subtasks as I start working, but a road map for now:

  • Autogenerate standard harnesses for types that already implement Arbitrary (either provided by Kani or implemented by the user)
  • Detect which functions have contracts and autogenerate contract harnesses for them.
  • Try to derive/implement Arbitrary for other types in the crate if doing so would allow us to verify more functions
  • Extend the harness autogeneration to support more complex use cases:
    • Types with invariants that the harnesses need to respect
    • Pointers
    • Loops
    • Generics
    • Harnesses for closures
  • Default timeout & loop unwinding bounds
  • Filters to include/exclude certain functions
  • Logging about which functions were skipped
@carolynzech carolynzech added the [C] Feature / Enhancement A new feature request or enhancement to an existing feature. label Jan 14, 2025
@carolynzech carolynzech self-assigned this Jan 14, 2025
@carolynzech
Copy link
Contributor Author

Note from #3874 -- Should log metadata about which functions were skipped, then print those functions in the driver, ideally with a note about why they were skipped.

github-merge-queue bot pushed a commit that referenced this issue Feb 11, 2025
This PR introduces an initial implementation for the `autoharness`
subcommand and a book chapter describing the feature. I recommend
reading the book chapter before reviewing the code (or proceeding
further in this PR description).

## Callouts
`--harness` is to manual harnesses what `--include-function` and
`--exclude-function` are to autoharness; both allow the user to filter
which harnesses/functions get verified. Their implementation is
different, however--`--harness` is a driver-only flag, i.e., we still
compile every harness, but then only call CBMC on the ones specified in
`--harness`. `--include-function` and `--exclude-function` get passed to
the compiler. I made this design choice to try to be more aggressive
about improving compiler performance--if a user only asks to verify one
function and we go try to codegen a thousand, the compiler will take
much longer than it needs to. I thought this more aggressive
optimization was warranted given that crates are likely to have far many
more functions eligible for autoverification than manual Kani harnesses.

(See also the limitations in the book chapter).

## Testing
Besides the new tests in this PR, I also ran against
[s2n-quic](https://github.com/aws/s2n-quic) to confirm that the
subcommand works on larger crates.

Towards #3832

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 and MIT licenses.
@carolynzech
Copy link
Contributor Author

carolynzech commented Mar 5, 2025

For now, we don't support automatic harnesses for closures for two reasons:

  1. We can't tell if a closure comes from Kani instrumentation (#[kanitool::is_contract_generated] attribute is not detected #3921).
  2. Generating an Instance from a closure CrateItem isn't working, i.e., the Instance::try_from(fn_item) call in #[kanitool::is_contract_generated] attribute is not detected #3921 fails on a closure like:
const MY_CLOSURE: fn(u8) -> u8 = |x: u8| x + 1;

whereas

const MY_CLOSURE: fn(u8) -> u8 = |x: u8| x + 1;

#[kani::proof]
fn harness() {
  MY_CLOSURE(kani::any());
}

runs fine--Kani finds the Instance for the closure just fine during its reachability analysis. I don't think this is necessarily worthy of a separate issue yet because it may just be something simple that I'm doing wrong, and it's a moot point as long as #3921 is still open.

github-merge-queue bot pushed a commit that referenced this issue Mar 6, 2025
Follow up to #3874
implementing the features requested during review. Hopefully reviewing
commit-by-commit makes the size manageable (most of the changes come
from the additional metadata logging) but happy to split this into
multiple PRs if not.

The list below maps each commit to the review comment on
#3874 that inspired it.

- 8685925:
#3874 (comment)
- f060d66:
#3874 (comment)
- 7a7f654:
#3874 (comment)
- 4f3add8:
#3874 (comment)
- 212b0ff:
#3874 (comment)
- f105a9c:
#3874 (review)
and
- 6a8dc61:
#3874 (review)
- 6f946bb and
6997afb:
#3874 (comment)

Towards #3832

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 and MIT licenses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[C] Feature / Enhancement A new feature request or enhancement to an existing feature.
Projects
None yet
Development

No branches or pull requests

1 participant