/
Automation Tests

Automation Tests

Owned by Harry Liu (Unlicensed)
Last updated: Nov 25, 2024 by Devin HebertVersion comment
5 min read

Our automation test scripts are in \src\MarineLMS.Test.QA . Test scripts are written by python.

We use TeamCity to auto-run our tests and keep record of results. TeamCity is a software which manages automation build and test tasks. It runs in a virtual machine manage by QA team. It fetches code from GitLab, create test tasks and run tests. You can also run test scripts locally, but your change in local will not affect what is in TeamCity unless it is pushed to correct branch in Git.

Run tests locally:

A guide to setup python env locally: https://rippleops.atlassian.net/wiki/spaces/QAT/pages/5432083107. Diproton is a framework that we created to simulate user behaviors when using the application. Even you are not using Diproton, you still need same setup to run your tests. (PS: Visual Studio has a issue that it sometimes cannot discover tests.) You can talk to QA team when you meet a problem and see if they have a solution. Usually this can be solved by re-install python env, clean bin folder, rebuild solution or reboot computer. Broken scripts may cause this issue as well. You can check if a script is broken by right click on the file and select start without debugging. It returns error message if broken.

Make sure you have correctly setup the Visual Studio before you start. With correct setup, you should to be able to see and run a list of tests in Test Explorer window after correct setup. If you cannot see all tests, there is something wrong with the setup.

The application needs to be running while you run the tests. You can right click on any test to either run or debug. If you want to debug a test script, the application needs to be in non-debug mode.

The “Start Without Debugging“ starts the application non-debug mode.

In test debug mode, the script pauses when it meets a break point.

 

TeamCity

TeamCity runs build and test procedures based on its config. Most of our tests are run weekly/nightly. NightlyBuild means tests are run every night. WeeklyBuild means tests are run every weekend. (you can view all past and running tasks in TeamCity)

Currently all ps-related tests are in Production QA tests - Nightly Build (develop branch). So every morning you need to check the result. If there are failed tests related to PS, you should create a task in Jira and let PS team know. Then inform QA team that PS team is working on it. The test will be muted so it will not keep sending alerts. When it is fixed, you need to let QA team know and let them un-mute the test.

You can click on the task and then click on following buttons (passed/muted/ignored) to view which tests are passed/muted/ignored

 

You can search ps to see all tests with ps in their names.

QA team is in charge if a test needs to be added(add a new test to TeamCity)/muted(the test will not send warning message when failed)/ignored(the test will not be run).

More TeamCity knowledge: https://rippleops.atlassian.net/wiki/spaces/QAT/pages/5432088880

 

Test scripts structure (code):

PS_APINightly_test:

(tests that run every night in TeamCity)

  • template tests:
    - Certificate Templates (compile and render the template) TODO: verify if all required resources (css, images) are in place
    - Notification Templates (compile and render the template)
    - NavPage Templates (compile) TODO: test rendering nav page templates

  • user importer tests: (each organization may have unique user importer setup, so we need to test them separately) TODO: create more test cases to cover more edge cases
    - BCF user importer
    - CMG user importer
    - CCL user importer
    - Default user importer

PS_Functional_test:

(tests that manually run)

  • Render certificate templates in browser (render the certificate, get the html string and save in local)TODO: render with template resources as well, currently template resources are not in place when testing

Tips on coding style:

  • Avoid “directly“ execute command (using script like root.API.executeCmd("")) in script page (i.e. notificationTemplates_test.py), since the command in system can be modified. Create a function to execute command. Collect all functions in a file in folder MarineLMS.Test.QA. (i.e. Template.py is a collection of all functions needed by template test script)

  • (TODO) Try to organize your test objects as classes. It helps you to protect data, manage you test steps and clean up after testing. (i.e. files in MarineLMS.Test.QA/LMSClasses)

  • Remember to delete generated files/folders at the end of your script

Here is code style guidelines document:
https://rippleops.atlassian.net/wiki/spaces/QAT/pages/5432088662

Procedure of creating automation script:

  1. Discuss with team about what to test, how it works and what are potential risks

  2. Ask QA team about where this script should belong to (weekly? nightly? manually?)

  3. Create a new branch, generate script file in corresponding folder (i.e. PS_Script/NightlyTests)

  4. Write script and create test files, try to keep all test files and generated files inside MarineLMS.Test.QA\assets or CustomerAssets (To avoid causing chaos)

  5. Create merge request in Git lab, ask for approve of merging into develop

  6. Ask QA team to enable the test in TeamCity

https://rippleops.atlassian.net/wiki/spaces/QAT/pages/5432083197

Procedure of automation test result check:

  1. Every morning you need to check the result.

  2. If there are failed tests related to PS, you should create a task in Jira and let PS team know.

  3. Then inform QA team that PS team is working on it. The test will be muted so it will not keep sending alerts.

  4. When the issue is resolved, you need to let QA team know and let them un-mute the test.

 

Ongoing Tasks: