8.1.3.1. Failure modes¶

• False-positive: A function is reported as covered, although it is not covered - Risk: Overreporting, could result in testing gap. - Mitigation: No mitigation, since we assume the likelihood of such an error low.

• False-negative: A function is reported as not covered, although it is covered - Risk: Underreporting, will not result in testing gap. - Mitigation: Since we want to achieve 100% line coverage this would stand out and be manually investigated.

• The code coverage instrumentation introduces bugs into the library or the test runner - Risk: That results in failing tests being reported as successful or successful tests being reported as failing - Mitigation: Running the test suite once with and once without code coverage instrumentation and ensuring both report the same result.

• Undercounting: Total number of functions is too high - Risk: A function is being considered, although it is not part of the certified subset - Mitigation: Not a risk as it only results in us testing more than necessary

• Overcounting: Total number of functions is too low - Risk: A function is not being considered, although it is part of the certified subset - Mitigation: Developing symbol-report which uses exactly the same information as the compiler

• Line that can be executed not being reported as executable

  • Risk: Underreporting, code that should be tested may not being tested

  • Mitigation:

    • blanket warns if a function has no executable line

    • (Future work) End-to-end test that ensures the correct lines are being reported as executable

8.2.2.1. Nightly features¶

The core library relies on a few so-called “nightly features” of the compiler. Regular users of Ferrocene are not allowed to use them, therefore they are not part of the compiler qualification. This is because they are either “experimental” or “internal”. They do work well, but they can change between compiler versions and do not fall under the usual Rust stability guarantees. This is not a problem for the core library, because rustc and the core library are developed, built, and tested together.

Nightly features are activated by setting the RUSTC_BOOTSTRAP environment variable when executing rustc.

Nightly features used by the core library are listed as #![feature(name_of_the_feature)] in library/core/src/lib.rs.

Nightly features are tested by the compiletest test suite, by tests that activate that feature explicitly. E.g. tests/ui/unknown-language-item.rs tests #![feature(lang_items)].

8.2.2.2. Nightly flags¶

The core library also relies on various normally unqualified flags of the compiler. These are only for the use of core; Ferrocene customers must still adhere to the permitted flags in Tool Options. Like nightly features, these flags work well but are experimental, internal, or simply have not yet been qualified. This is not a problem for the core library, because rustc and the core library are developed, built, and tested together.

The flags used for core are enabled by passing them to the compiler when the core library is being built.

The flags used for core are directly tested by the compiletest test suite, by tests that pass that flag explicitly. Flags are also indirectly tested each time the core library is used, and the core library is used extensively by the compiler in all test suites.

8.2.2.3. Compiler built-in functions¶

There are functions in the core library that are “compiler built-in”. That means they are not implemented in the core library codebase but in the compiler codebase.

They fall in two categories:

8.3.3.1. Failure modes¶

• False-negative: Fail to detect non-compliance with the consistency rules

  • Risk: Diverging from consistency rules. This is not critical, because Ferrous Systems only consumes the code from upstream and does not impose additional rules on it.

  • Mitigation: None. If found, report issue upstream.

• False-positive: Report non-compliance, although the code is compliant

  • Risk: None

  • Mitigation: Report issue upstream.

8.4.3.1. Failure modes¶

• False-positive: Report test as successful, although it is failing

  • Risk: Not detect incorrect code.

  • Mitigation: Report issue upstream.

• False-negative: Report test as failing, although it is successful

  • Risk: None

  • Mitigation: Report issue upstream.

8.5.3.1. Failure modes¶

• Test/dist mismatch: The core library is tested in a configuration that does not match the distributed code.

  • Risk: All tests pass, but the distributed code would have failed the tests.

  • Mitigation: Package exactly the code that was tested.

• Incorrect configuration: The core library is built in a configuration that does not match the safety plan.

  • Risk: Some certified code calls uncertified code.

  • Mitigation: Building for a certified target automatically sets configuration flags that ensure all certified code only calls certified code.

    Tests and coverage collection run on that certified target.

• Missing build flags: The core library is built without flags required by the safety plan, such as -Cpanic=abort. - Risk: The core library uses unwinding panics at runtime, which are not certified. - Mitigation: The tests for the core library, when instrumented for coverage, give a compilation error if the panic = "abort" conditional compilation option is unset.

• Unqualified flags: The core library is built with extra flags that are not qualified, such as -Z inline-mir. - Risk: Unqualified compiler flags introduce errors. For example, MIR inlining introduces a codegen bug. - Mitigation: Refer to Nightly flags.

8.6.3.1. Failure modes¶

• Missing test job: A test job fails or is not run, but the core library is distributed to customers anyway.

  • Risk: Customers run untested code.

  • Mitigation: All “distribution” CI jobs depend on test jobs.

    If a test job fails, the distribution job never runs.

• Missing test suite: A test suite is never run in any test job. - Risk: Not all test suites in the safety plan are run. - Mitigation: The ferrocene/ci/split-tasks.py script chooses which suites are run in which job.

  This script has an exhaustive list of all test suites required by our safety plan, and ensures each suite is run in at least one job.

8.7.3.1. Failure modes¶

• False-positive: Introduce changes that were not made

  • Risk: Erroneous code, documentation, configuration

  • Mitigation: Code review.

• False-negative: Do not track changes that were made

  • Risk: Lose time invested.

  • Mitigation: Code review.

8.8.3.1. Failure modes¶

• Modify generated documentation

  • Risk: Erroneous documentation

  • Mitigation: If detected, report error.