Help:Pickle/Glossary



Glossary for pickle tests is a list of core terms and concepts, both for 1>#spec|#spec and 2>#step|#step style tests. Terms are borrowed from the ITIL, ISTQB, RSpec framework, and predicate logic. The different frameworks are linked as necessary. Try to keep close to the definitions, but adapt to this testing environment.

Terms and concepts

 * After
 * Marks functions to be run after 1>#describe|#describe, 2>#context|#context, and 3>#it|#it. It uses a single queue so all functions will be run in the correct sequence.
 * Part of the teardown of 1>#test fixture|#test fixture.
 * Also known as teardown.


 * AfterAll
 * Marks functions similar to 1>#after|#after, except it is run only once. Strictly speaking this function is not necessary as code can be added inline.  It use a single queue so all functions will be run in the correct sequence.  It use the same queue as after so it will be run at the correct time in the first test.  At later invocations it will not be run.
 * Part of the teardown of 1>#test fixture|#test fixture.


 * Around
 * An alternate to 1>#after|#after that is only run if the test 2>#throw|#throws an 3>#exception|#exception. If an exception is thrown, and there is an around function registered, then no after function will be run.
 * Part of the teardown of 1>#test fixture|#test fixture.


 * Assertion
 * Test that focuses on the actual code, and assertions about that code, and their falsy 1>#condition|#conditions.
 * Part of the verification of 1>#test fixture|#test fixture.
 * Wikipedia: Assertion


 * Before
 * Marks functions to be run before 1>#describe|#describe, 2>#context</>|#context, and 3>#it</>|#it. It use a single queue so all functions will be run in the correct sequence.
 * Part of the setup of 1>#test fixture</>|#test fixture.
 * Also known as setup.


 * BeforeAll
 * Marks functions similar to 1>#before</>|#before, except it is run only once. Strictly speaking this function is not necessary as code can be added inline.  It use a single queue so all functions will be run in the correct sequence.  It use the same queue as before so it will be run at the correct time in the first test.  At later invocations it will not be run.
 * Part of the setup of 1>#test fixture</>|#test fixture.


 * Carp
 * A 1>#spy</>|#spy that adds a message to stack without exiting the test, printing the caller's name and its arguments. Partly emulating the <tvar|1> </> function from Perl.
 * Perldoc: Carp


 * Cluck
 * A 1>#spy</>|#spy like 2>#carp</>|#carp, but also prints a stack trace starting one level up. Partly emulating the <tvar|1> </> function from Perl.
 * Perldoc: Carp


 * Condition
 * A logical expression that can be evaluated as True or False, e.g., A>B.
 * Part of the 1>#assertion</>|#assertion.
 * ISTQB Glossary: Condition


 * Confess
 * A 1>#spy</>|#spy like 2>#croak</>|#croak, but also prints a stack trace starting one level up. Partly emulating the <tvar|1> </> function from Perl.
 * Perldoc: Carp


 * Context
 * Marks functions used as examples, but also a function <tvar|1> </> itself.
 * See 1>#example</>|#example for details.


 * Continuous integration
 * Agile approach to software development to minimize the duration and effort during each iteration, and at the same time deliver a software suitable for release.
 * This dual objective requires an integration procedure which is reproducible, usually by automated integration and builds – often multiple times each day, and achieved through extensive testing, version control, team policies and conventions.
 * Agile Alliance: Continuous Integration
 * Wikipedia: Continuous integration


 * Coverage
 * The degree, expressed as a percentage, to which a specified coverage item has been exercised by a test suite.
 * ISTQB Glossary: Coverage


 * Criteria
 * Decision rules used to determine whether a test item (function) or feature has passed or failed a given test. In specs it is only used simple pass/fail.
 * ISTQB Glossary: Pass/fail criteria


 * Croak
 * A 1>#spy</>|#spy like 2>#carp</>|#carp, but also stops the running test (the user provided anonymous function). Partly emulating the <tvar|1> </> function from Perl.  Because it throws an exception it will always trigger a stack trace.
 * Perldoc: Carp


 * Describe
 * Marks functions used as examples, but also a function <tvar|1> </> itself. See 1>#example</>|#example for details.


 * Design
 * Also known as test design
 * The process of transforming general test objectives into tangible test conditions and test cases.
 * The document that describes the implementation details of the test, or results of, the system or in our case the module. The document is a subsection on the   subpage for the spec page.
 * ISTQB Glossary: Test design


 * Developer
 * A person that write code. See 1>#tester</>|#tester


 * Documentation
 * Also known as test documentation
 * The document that describes plans of, or results of, the system or in our case the module. The document is the   subpage for the spec page.


 * Dummy
 * A necessary minimum implementation of functionality that is only necessary to make some other implementation available for use. Such objects are usually created inside the tests, and lifespan is limited to the tests.  A dummy is a type of 1>#test double</>|#test double.
 * Wikipedia: Dummy


 * Duration
 * Time elapsed while executing a test, test run, or the entire test execution of a build or release pipeline. Scribunto uses cpuLimit to limit total duration of a test run.
 * Scribunto configuration


 * Examples
 * The levels in the 1>#describe</>|#describe, 2>#context</>|#context, and 3>#it</>|#it ladder. The name and level are a bit arbitrary as they all have nearly the same function.  The context is often left out, and only describe and it are used.


 * Exceptions
 * Error states reported from within the code. They will be caught by the test framework.
 * Wikipedia: Exception handling


 * Expectations
 * Tests that focus on the provided objects and examples, and their truthy states.


 * Fail
 * A test is deemed to fail if its actual result does not match its expected result.
 * ISTQB Glossary: Fail


 * Fake
 * A simpler implementation of functionality that otherwise would be to heavy or difficult to implement or use. Such objects are usually created inside the tests, and lifespan is limited to the tests.  A fake is a type of 1>#test double</>|#test double.
 * Wikipedia: Fake object


 * Feature
 * An attribute of a component or system specified or implied by requirements documentation (for example reliability, usability, or design constraints).
 * ISTQB Glossary: Feature


 * Fixture
 * The function defining the test is the test fixture, and a frame without a test fixture it will be skipped.
 * Wikipedia: Test fixture


 * Good
 * See 1>#pass</>|#pass for details.


 * Harness
 * A test environment comprised of stubs and drivers needed to execute a test.
 * ISTQB Glossary: Harness


 * It
 * Marks functions used as examples, but also a function <tvar|1> </> itself. See 1>#example</>|#example for details.


 * Mock
 * A simulated object that act as a strictly controlled replacement for a real object. A mock is a type of 1>#test double</>|#test double.
 * Agile Alliance: Mock Objects
 * Wikipedia: Mock object


 * Overfitting
 * Happens when there are too tight couplings between the test and source code. Usually this happen when tests use spies to verify internal data structures in source code.


 * Pass
 * A test is deemed to pass if its actual result matches its expected result.
 * ISTQB Glossary: Pass


 * Pending
 * A frame marked with 1>#skip</>|#skip or 2>#todo</>|#todo will be in a pending final state. This state is an override of 1>#pass</>|#pass ("ok") and 2>#fail</>|#fail ("not ok") and makes it possible to code and test with less noise.


 * Return
 * The outermost zero-level that encapsulates the test results and returns them in a formatted fashion.


 * Setup
 * See 1>#before</>|#before for details.


 * Skip
 * The system (or optionally the author) can mark a test as skipped, either in the description or in the code. Further processing within the frame will then be terminated and the current state used.


 * Spec
 * A type of testing to make sure we build the thing right, usually by writing some kind of unit tests, but can also be more high level.


 * Spies
 * Functions that can be registered on other functions, or injected into code to report or alter internal states. Spies are one of 1>#carp</>|#carp, 2>#cluck</>|#cluck,  3>#croak</>|#croak, and 4>#confess</>|#confess.  Spying on public calls made before the module is available to the testing regime will not be possible.  A spy is a type of 1>#test double</>|#test double.


 * Step
 * A type of testing to make sure we build the right thing, usually by writing some kind of acceptance tests, which is at the integration level.


 * Stub
 * Simulated code, usually a function in this context, that acts as a strictly controlled replacement for some real code. A stub is a type of 1>#test double</>|#test double.
 * ISTQB Glossary: Stub
 * Wikipedia: Test stub


 * Subject
 * The object under test. It can be explicitly set or set as part of the 1>#examples</>|#examples.  It is passed on to 1>#expect</>|#expect.


 * Teardown
 * See 1>#after</>|#after for details.


 * Test
 * An activity to verify proper operation of a system, given that it is executed under specific conditions, and with observed and recorded results. Consists of one or more 1>#test case</>|#test cases.
 * ISTQB Glossary: Test case


 * Test case
 * A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. Can also include documentation for how to run the test case.
 * ISTQB Glossary: Test case


 * Test doubles
 * Code replacing pieces of code that are not tested, and should return only known values.
 * Wikipedia: Test double


 * Test item
 * The individual element to be tested. Usually it is one test object and many test items.
 * ISTQB Glossary: Test item


 * Tester
 * A skilled user who is involved in the testing of a component or system. See 1>#developer</>|#developer
 * ISTQB Glossary: Tester


 * Throw
 * The test code can run into a statement that either can't be executed and thus "throws" an exception or explicitly "throws" an exception at that point. When the code throws an exception further execution is stopped and a report is initiated.


 * Todo
 * The user (or optionally the system) can mark a test as todo, either in the description or in the code. Further processing within the frame will continue, but the final state set to pending anyhow.