python pytest selenium requests PyCharm pylint

license

Test automation framework for testing UI, front-end and back-end API of restful-booker web-site - https://automationintesting.online.

2.1. Table of contents

  1. Resources of web app

    1. Restful-booker app resources

      1. Restful-booker web-site URL

      2. Restful-booker API

    2. Booking side

      1. Home web page

        1. Basic structure of Home web page

      2. Booking web page

    3. Admin Side

      1. Admin (Rooms) Web Page

      2. Report web page

      3. Branding web page

      4. Messages web page

  2. Test framework configuration and setup

    1. Required python packages

      1. For tests itself

      2. For UI/Web testing

      3. Utils/configs

      4. Logging/reporting

      5. Data generators

      6. DB

      7. Assertions

      8. Code analysing

  3. Project docs

  4. Project setup process

    1. Setup by scripts

    2. Create and activate a new virtual environment

    3. Install required packages from requirements.txt

    4. Check prerequisites for Allure test report generation

    5. Usage DB as source of test data

      1. Update DB content

    6. Checking project dependency tree

      1. pipdeptree

    7. Generating documentation by Sphinx

  5. Tests

    1. Test execution by pytest

    2. Run all Tests

    3. Run tests in a module

    4. Test run for specific test (by test name) by command

    5. Run Tests in a directory

    6. Run tests of a specific class

    7. Run tests by marker expressions

    8. Run tests by node IDs

    9. Additonal parts for test run/configuration

      1. Specifying configuration file location

      2. Rerun failed tests

      3. Run tests in parallel(using workers)

      4. Snippets for running existing tests

    10. Test report generation

      1. By pytest (Simple Report)

      2. By pytest (HTML Report)

      3. By pytest (JUnit XML Report)

      4. Test reports by Allure

        1. Creating initial pytest reports by Allure

    11. Existing(created) tests by categories

      1. Test List

  6. Publishing Sphinx doc on GitHub pages

  7. Resources for TA frameworks

2.2. Resources of web app

2.2.1. Restful-booker app resources

2.2.1.1. Restful-booker web-site URL

https://automationintesting.online/

2.2.1.1.1. Restful-booker API

https://restful-booker.herokuapp.com/apidoc/index.html

2.2.2. Booking side

go to TOC

2.2.2.1. Home web page

go to TOC.

Home web page: https://automationintesting.online

restful-booker_site__home_page_main.png

2.2.2.1.1. Basic structure of home web page

```
    logo(picture)
        welcome_text
    rooms_section
        room_section
            hotel_picture
            hotel_room_description
                hotel_room_description_header
                    hotel_room_type
                    hotel_room_wheelchair_option
                hotel_room_description_title
                hotel_room_options
                    hotel_room_option_wifi
                    hotel_room_option_refreshments
                    hotel_room_option_tv
                    hotel_room_option_safe
                    hotel_room_option_radio
                    hotel_room_option_views
                hotel_room_booking_button
            booker_contact_form
                name_input_form
                email_input_form
                phone_input_form
                subject_input_form
                message_input_form
                submit_button_on_contact_form
                alert_booker_form
            hotel_contact_details_field
            map_picture
            footer_panel
                site_owner_info
                cookie_policy_link
                privacy_policy_link
                admin_panel_link
```

2.2.3. Booking web page

go to TOC.

Booking web page: https://automationintesting.online

Same URL as for Home page but with additional components on the page

restful-booker_site__home_page_main.png

2.2.4. Admin side

2.2.4.1. Admin (Rooms) web page

go to TOC.

Home web page: https://automationintesting.online/#/admin/

restful-booker_site__admin-room_page.png

2.2.4.2. Report web page

go to TOC.

Report web page: https://automationintesting.online/#/admin/report

restful-booker_site__report_page.png

2.2.4.3. Branding web page

go to TOC.

Branding web page: https://automationintesting.online/#/admin/branding

restful-booker_site__branding_page.png

2.2.4.4. Messages web page

go to TOC.

Messages web page: https://automationintesting.online/#/admin/messages

restful-booker_site__messages_page.png

2.3. Test framework configuration and setup

2.3.1. Required python packages

go to TOC.

In this project used ‘pip-tools’ and approach to use high-level dependency tree. Main required packages presented in requirements.in. All used Python packages for the current project are generates in requirements.txt

Below is the list of main packages with references

2.3.1.1. For tests itself

go to TOC.

2.3.1.2. For ui/web testing

go to TOC

2.3.1.3. Utils/configs

go to TOC

2.3.1.4. Logging/reporting

2.3.1.5. Data generators

go to TOC

2.3.1.6. DB

2.3.1.7. Assertions

go to TOC

2.3.1.8. Code analysing

2.4. Project docs

This project contains generated documentation by Sphinx All documentation you can find by opening index.html in the browser or check ta_framework_ui_api.pdf Sphinx setup and generating process described here

2.5. Project setup process

go to TOC.

2.5.1. 1. Setup by scripts

By default, you can use file:

  1. setup_env.bat for Windows-based machine or

  2. setup_env.sh UNIX-based for validating python version, installing all packages required for running current project. Otherwise, you can manually install everything using described steps below or fix some issues that appears while project was configured via scripted file.

2.5.2. 2. Create and activate a new virtual environment:

go to TOC.

  • Create virtual environment.
    Script below is creating environment with name ‘env’.
    If you want to create environment with unique name, please replace the env name using your env name in script
    python -m {here_is_your_venv_name} ../env

    Working script for creating venv with name ‘venv’ is below:

    python -m venv ../.venv

    then activate it

    • for unix-based

    source ../venv/bin/activate

    • for windows

    .\.venv\Scripts\Activate

    If you like to have different name for the environment

    python -m venv {venv_for_project}

    and then

    source {venv_for_project}/Scripts/activate

For deactivating created env use command * shell     deactivate    

2.5.3. 3. Install required packages from requirements.txt

go to TOC.

pip install -r requirements.txt

If requirements.txt file is missing request, or you have different configuration of the project after installation, please check generate new requirements.txt file using command

pip-compile requirements.in

Created requirements.txt file will have all dependency for the project.

In case if you are using ‘pip-tools’ do next steps,

  1. check requirements.in content for preventing conflicts with existing(venv/global) configurations.

  2. compile requirements.txt by

pip-compile requirements.in

  1. install dependencies by

pip install -r requirements.txt

2.5.4. 4. Check prerequisites for Allure test report generation

go to TOC.

  1. Check your system on installed and available :

    • JAVA

      java -version

    • Node.js

      node -v

      If programs are missing install them using info below

  2. Install Java 3

    • Download Java 3 HERE

  3. Install Node.js

    • Download Node.js HERE

  4. In command prompt, run the below command

    npm install -g allure-commandline

  5. Add npm and allure-commandline to system path

    %AppData\Roaming\npm

  6. Check that system can have access to allure by

    allure --version

  7. Install allure-pytest

    pip install allure-pytest

    or from your IDE

2.5.5. 5. Usage DB as source of test data

go to TOC.

In case if your test data is stored in DB or external files, check the project configuration and adjust test data manually.

2.5.5.1. 1. Update DB content

go to TOC.

If you are using data from DB generate data based on your testing model or paste it manually preferable to use pip install mysql-connector-python

2.5.6. 6. Checking project dependency tree

2.5.6.1. pipdeptree

Basic Dependency Tree

pipdeptree

Dependency Tree in JSON Format

pipdeptree --json

Dependency Tree with Outdated Packages

pipdeptree --outdated

Dependency Tree for a Specific Package

pipdeptree -p <package_name>

Generate a Text Report and Save to File

pipdeptree > resources/project_dependencies.txt

Visualize Dependency Tree with Graphviz first install graphviz:

pip install graphviz

then generate a graphical representation:

pipdeptree --graph-output png > resources/project_dependencies.png

2.5.7. Documentation

2.5.8. Pydocstyle

To check your code for missing docstrings for current project selected pydocstyle.

pydocstyle core

2.5.8.1. 3. Check an Entire Directory

To check all Python files in a specific directory, you can run:

bash pydocstyle path/to/your_directory/

2.5.8.2. 4. Check with Specific Conventions

You can specify the docstring conventions you want to enforce using the --convention flag. The available conventions are:

  • google

  • numpy

  • pep257

For example, to check using the Google style:

bash pydocstyle --convention=google path/to/your_directory/

2.5.8.3. 5. Ignore Specific Errors

If you want to ignore certain error codes, you can use the --ignore option followed by the error codes (comma-separated). For example, to ignore error code D103 (missing docstring in public function):

bash pydocstyle --ignore=D103 path/to/your_directory/

2.5.8.4. 6. Output Format

You can change the output format using the --format option. The default format is a simple text output, but you can change it to json for easier parsing:

bash pydocstyle --format=json path/to/your_directory/

2.5.8.5. 7. Verbose Mode

To see more detailed output, you can run pydocstyle in verbose mode:

bash pydocstyle --verbose path/to/your_directory/

2.5.8.6. 8. Checking a Specific File or Line

You can check a specific line in a file by appending the line number to the file path. For example, to check line 42 of your_file.py:

bash pydocstyle path/to/your_file.py:42

2.5.8.7. 9. Show Available Error Codes

To see a list of all available error codes and their meanings, use the --help option:

bash pydocstyle --help

2.5.8.8. Summary of Commands

  • Check a specific file:

    pydocstyle path/to/your_file.py

  • Check an entire directory:

    pydocstyle path/to/your_directory/

  • Specify conventions:

    pydocstyle --convention=google path/to/your_directory/

  • Ignore specific errors:

    pydocstyle --ignore=D103 path/to/your_directory/

  • Output in JSON format:

    pydocstyle --format=json path/to/your_directory/

2.5.8.9. Generating documentation by Sphinx

  1. Install Sphinx in your project environment:

pip install sphinx

  1. Set Up Sphinx in Your Project !!! IMPORTANT For more convenient usage of project structure to better create separate folder where will be stored all documentation and related Sphinx config files and folder.
    For this project was created new folder docs and from that place all Sphinx related command should be executed. After navigation to the docs directory, run:

    sphinx-quickstart

This command will guide you through setting up Sphinx by asking several configuration questions.

Typical responses:

  • Separate source and build directories: Yes

  • Project name: [Your project name]

  • Author name: [Your name]

  • Project version: [Your project version]

  • Project language: [en]

    This will generate
    1. source/ directory with a default conf.py file for configuration and some starter .rst (reStructuredText) files.
    2. build/ directory (will be empty). It’s default directory that will have all project related data if you will not use any different name for collecting generated Sphinx docs (pdf, html… etc).
    It can be deleted or renamed if you want to have separate places for different types of generated docs. For this project will be generated documentation in HTML and PDF formats. Below you will find scripts and instructions for generating documents in PDF and HTML format using different Sphinx extensions. 3. make.bat - default script file for generating docs on Windows OS 4. Makefile - default script file for generating docs on UNIX base OS

  1. Configure conf.py Edit the generated conf.py file located in the source/ directory to customize your documentation. Key configurations to include:

    • Extensions: Enable useful Sphinx extensions, such as autodoc for auto-generating documentation from your Python docstrings:

    extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.napoleon',  # For Google-style or NumPy-style docstrings
]

  • Paths: Set the path for your Python modules to be included in the docs:

    import os
import sys
sys.path.insert(0, os.path.abspath('../..'))

  • HTML Theme: You can set the theme for your HTML documentation (default is alabaster):

    html_theme = 'sphinx_rtd_theme'  # Example: ReadTheDocs theme

  • Source file suffix*: Define which file extensions to look for:

    source_suffix = ['.rst', '.md']

  1. Install related to chosen Sphinx extensions packages.

    1. For ‘sphinx_rtd_theme’ theme execute

      pip install sphinx_rtd_theme

    2. For ‘myst_parser’ theme execute

      pip install myst_parser

    3. For ‘rst2pdf’ theme execute

      pip install rst2pdf

  2. Document Your Code.

  3. Generate reStructuredText (.rst) Files To automatically generate .rst files from your Python code, use sphinx-apidoc. This will generate a source/ directory with .rst files that describe your modules and classes:

    sphinx-apidoc -o source/ path/to/your/module

    !!! IMPORTANT
    For the current project all Sphinx documentation is stored in docs folder and script below should be executed from the project root folder

    sphinx-apidoc -o docs/source .

  4. Build HTML Documentation

    !!! IMPORTANT
    For the current project script below should be executed from the project root folder.
    Once your .rst files are in place and conf.py is configured, you can build the HTML documentation:

    • on Windows OS

      sphinx-build -b html docs/source/ docs/html/

      where html_docs - custom defined name for storing generated html docs.

      If errors appears after generating docs try to clear the cached build.
      Clear the build/ directory by deleting it or running:

      sphinx-build -b html -E source/ html/

      The -E flag forces Sphinx to rebuild everything from scratch, avoiding potential caching issues.

    • on UNIX

      make html

      This command will generate the HTML files inside the build/html/ directory. Open the index.html file in your browser to view your documentation.

  5. Build PDF Documentation

  • on Windows OS

    sphinx-build -b pdf source/ pdf_docs/

  1. View Documentation

Navigate to the build/html/ directory and open index.html in your browser to view your generated documentation.

Additional Configurations (Optional) Include Markdown files: If you want to include .md files, ensure the myst_parser is installed and enabled:

	pip install myst-parser

Then add it to conf.py:

	extensions = ['myst_parser']

Customizing Themes: You can install additional Sphinx themes by running pip install and configuring html_theme in conf.py.

2.6. Tests

go to TOC.

More information about test runs by pytest you can find here

2.6.1. Test execution by pytest

go to TOC.

All tests are located in tests folder

2.6.1.1. Run all Tests

go to TOC.

To run all the tests from the root directory, you can use the following command:

python pytest

or just

pytest

You can add the -v flag to get more verbose output:

python pytest -v

python_pytest_verbose.png

You can enable live console logging using the pytest -s command too.

2.6.1.2. Run tests in a module

go to TOC.

To run all tests in a specific file (module), use the following command:

pytest tests/unit/test_functions.py

2.6.1.3. Test run for specific test (by test name) by command

go to TOC.

Use the -k option followed by the name of the test function or method you want to run

python -k {some_test}} e.g. python -k test_dummy_test.py

2.6.1.4. Run tests in a directory

go to TOC.

Perhaps you may decide to split your tests by unit, integration, end-to-end, performance, regression and so on.
If you need an overview of the various types of testing for your Python applications, this article on the types of
software testing is a good introduction.
In these cases it’s helpful to run tests within a specific directory, and you can use:

pytest {path_to_the_folder_with_test}

e.g.

pytest tests/dummy_tests

2.6.1.5. Run tests by node IDs

go to TOC.

To run a specific test, you can use the test’s node ID, which is essentially its path in the syntax:
{filename.py}::{test_function_name}.
For example, to run the test_add_negative_numbers function in the test_functions.py file, you can use the following command:

pytest  test/dummy_tests/test_functions.py::test_add_positive_numbers

This runs the test_add_positive_numbers test in the test_functions.py file.

2.6.1.6. Run tests of a specific class

go to TOC.

You can also run all tests in a specific class. To do this, you use the :: operator followed by the class name.
For example, to run all tests in the RegressionTests class, you can use the following command:

pytest test/dummy_tests/test_functions.py::TestsUnit

2.6.1.7. Run tests by marker expressions

go to TOC.

By using markers, you can run specific groups of tests, exclude tests, and prioritize tests. This can help you to write better tests and to get more value from your test suite.

We’ve covered several kinds of markers in the articles on Pytest Timeout, Pytest Skip Tests and Pytes Asyncio.

In Pytest, you can assign markers to your test functions using the @pytest.mark decorator. You can then use these markers to run specific tests.

This is especially useful when you have different types of tests, such as fast and slow tests, and you want to run them selectively.

Example of pytest written with markers presented below

@pytest.mark.unit
def test_one():
    result_of_doing = do_something()
    assert result_of_doing ==1

To run tests based on marker expressions, you use the -m flag followed by the marker name.

[pytest]
markers =
    unit : unit tests
    end_to_end  : end to end tests
    skip : slow tests`

command is

pytest -m unit

In case if ini file located not in the root directory marker should be passed via flag -c and valid configured path to the ini file

2.6.2. Additional parts for test run/configuration

2.6.2.1. Specifying configuration file location

go to TOC.

The -c option allows you to specify the path to your pytest.ini file directly.

pytest -c path/to/your/pytest.ini
example of usage for current project is pytest -c config/pytest.ini -m unit

Better to set the PYTEST_ADDOPTS environment variable to include the config file path:

export PYTEST_ADDOPTS="-c config/pytest.ini"

2.6.2.2. Rerun failed tests

Rerun for failed tests work after installation of pytest-rerunfailures

pip install pytest-rerunfailures

Check version of pytest-rerunfailures

pytest-rerunfailures --version

pytest --reruns 3 --alluredir="resources/project_test_reports/allure_reports

or in paralel

pytest  -n 10 --reruns 3 --alluredir="resources/project_test_reports/allure_reports"

Full Command Breakdown

-n 10: Runs tests in parallel using 10 worker processes.

--reruns 3: Reruns any failed tests up to 3 times.

--alluredir="resources/project_test_reports/allure_reports": Specifies the directory to store Allure reports.

2.6.2.3. Run tests in parallel(using workers)

Installing Required Plugins

Make sure you have the required plugins installed:

  • for parallel test executionpytest-xdist

pytest -n 3

-n 4: Runs tests in parallel using 4 worker processes.

2.6.2.4. Snippets for running existing tests

go to TOC.

  1. Test for checking home page (not a booking view)

pytest test/web_app_tests/test_login_page.py::test_check_main_section_of_home_page

2.6.3. Test report generation

go to TOC.

2.6.3.1. By pytest (Simple Report)

Simple pytest report generation in html format is possible by executing command

Short Traceback:

pytest --tb=short

Long Traceback (default):

pytest --tb=long

No Traceback (only show test results):

pytest --tb=short --disable-warnings

2.6.3.2. By pytest (HTML Report)

pytest --html=resources/project_test_reports/pytest_reports/pytest_general_test_report.html

using workers

pytest -n 10 --html=resources/project_test_reports/pytest_html_reports/pytest_html_general_test_report.html

2.6.3.3. By pytest (JUnit XML Report)

pytest --junitxml=resources/project_test_reports/pytest_junit_xml_reports/pytest_junit_xml_general_test_report.xml

2.6.3.4. Test reports by Allure

go to TOC.

2.6.3.4.1. Creating initial pytest reports by Allure

go to TOC. For generating Allure reports on the tests performed, you must first execute tests by pytest. You need specify a path for the test results directory in the –alluredir command-line argument when running your tests.

pytest --alluredir="resources/test_report/allure_reports"

If any any error appears

  • Check that the system statisfy Allure requirements.

  • Confirm that all components installed, report folder is created. need to specify execution tests by pytests with specific output format.

When test execution completed , Allure reports generation starts after executing

allure serve {path_to_report_folder}

Example of command execution:

(.venv) {project_dir_path} allure serve "resources/allure_reports"

command that is applicable for the current project configuration is

allure serve "resources/test_report/allure_reports"

After successful execution of command allure starts server and provide server URL for checking allure report in the browser.

2.6.4. Existing(created) tests by categories

go to TOC.

2.6.4.1. Test List

List of existing tests for the current project is possible to generate by executing python script make_list_of_tests.py. Result will be presented in the list_of_all_project_tests.md.
All tests are grouping by categories.
IMPORTANT!!! If you did any changes, please validate path to the test dir in the make_list_of_tests.py

2.7. Publishing Sphinx doc on GitHub pages

go to TOC.

  1. Generate Sphinx HTML Docs (If docs were not created, see section Generating documentation by Sphinx)

  2. Create a Separate Branch for Documentation (Optional but Recommended):

    • You can store the generated documentation in a dedicated branch, such as gh-pages, to keep it separate from your main project code

      git checkout --orphan gh-pages
git rm -rf .

    • Add the generated HTML files:

      cp -R docs/_build/html/* .
git add .
git commit -m "Publish Sphinx docs"
git push origin gh-pages

  3. Set Up GitHub Pages:

    • Go to your repository settings on GitHub.

    • Under Pages, select the branch (e.g., gh-pages) and folder (e.g., /root or /docs) where your documentation is stored.

    • Save the settings, and GitHub will publish your documentation at a URL like:

    https://<username>.github.io/<repository-name>/

  4. Automate Documentation Updates (Optional): Use a CI/CD pipeline (e.g., GitHub Actions) to regenerate and deploy the docs whenever you push updates to your repository. Here’s an example workflow:

  5. Latest working config is below

name: Deploy Sphinx Docs on GitHub pages

on:
  push:
    branches:
      - master  # Change to your default branch

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'

      # This step is skipped since you don't need to install dependencies
      # - name: Install dependencies
      #   run: |
      #     pip install sphinx
      #     pip install -r docs/requirements.txt

      - name: Build documentation
        run: sphinx-build docs docs/_build/html

      - name: Deploy to GitHub Pages
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: docs/_build/html

This config also published only README.MD

name: Deploy Sphinx Docs on GitHub pages

on:
  push:
    branches:
      - master  # Change to your default branch

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'

      - name: Install dependencies
        run: |
          pip install sphinx
          pip install -r docs/requirements.txt

      - name: Build documentation
        run: sphinx-build docs docs/_build/html

      - name: Deploy to GitHub Pages
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: docs/_build/html

GitHib workflow config that just render README.MD file and not shows exact Sphinx docs is below

name: Deploy Sphinx Docs

on:
  push:
    branches:
      - master  # Trigger on pushes to the main branch

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.9'

      - name: Install dependencies
        run: |
          pip install -r requirements.txt
          pip install sphinx

      - name: Build Sphinx documentation
        run: sphinx-build docs docs/_build/html

      - name: Deploy to GitHub Pages
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: docs/_build/html

2.8. Resources for TA frameworks

go to TOC.