Time Machine Test Runs

Description
kaitlyn.opperman
Posts: 8
Joined: Mon Apr 28, 2014 7:08 pm

Time Machine Test Runs

Postby kaitlyn.opperman » Wed May 21, 2014 12:22 pm

We currently use Polarion 2013 SR3 for requirements management and release records. We are now trying to extend our usage to our test management as well.
Our goal is to use a LiveDoc Document to capture the manual steps we have to run, and release this document and the related tests. We then need to be able to rerun the same manual steps at any time to reproduce the results.

One of the biggest Configuration Management concerns is the ability to reproduce the tests as they were run. If I create a baseline, I can use the "Time Machine" baseline viewer (as of Polarion 2012) to see all of the captured test runs and to see the manual test steps as they were captured. This is all very close to what we need. While the document and tests are clearly captured in the baseline, and all test steps are locked down, all test runs are also locked down and unable to be "run" again at this revision. There seems to be no way in the baseline view to re-run or create a new run. I understand the baseline is supposed to be read only, similar to a tag in subversion, but it would be really great to find a way to rerun this in Polarion instead of having to print out all of the test steps and manually fill in the steps to reproduce the run. This is still the closest I have found so far to what we need.

The other option I tried was to open the LiveDoc at a specific revision (the one released) and run the tests captured here. The problem with this technique is that the test runs will always populate with the latest version of the filtered work items, so if any steps have changed since the released revision, this new version of the test is shown instead of the steps from the LiveDoc revision.

Is there any way to rerun (in Polarion) a baseline/specific revision of all tests for a test run with the locked down steps?

Ideally this new run would create a new test record but display and record the old steps. I guess the best way to describe this would be similar to branching the released revision in subversion, running the test tun, and merging back only the new test records (and possibly the test run).

Sandro_F
Posts: 19
Joined: Wed Jan 19, 2011 8:40 am

Re: Time Machine Test Runs

Postby Sandro_F » Thu May 22, 2014 10:15 am

Well, you can execute the test run for the document in "Tree-View" with specific document name and revision id:

Code: Select all

/polarion/#/project/test_03/wiki/00_Documents/My_Testplan_Document?query=(*:*%20AND%20NOT%20type:heading%20AND%20TEST_RECORDS:(%22test_03/testrun1%22,*)%20AND%20NOT%20TEST_RECORDS:(%22test_03/testrun1%22,@any))&tree_depth=0&tab=tree&revision=300375


You will see the document work items for the specific document revision and can also "retest" the test cases.

kaitlyn.opperman
Posts: 8
Joined: Mon Apr 28, 2014 7:08 pm

Re: Time Machine Test Runs

Postby kaitlyn.opperman » Tue May 27, 2014 4:05 pm

Sandro_F,

Thank you for the response. This seems to be close to what I had hoped. I still do not quite understand how to do this in a more automated way.
Is the intended method manually selecting the tests in the document and then executing from the document tree view?
Is there a way for the test run to know it should be using an older revision without going through the Live Document hsitory?
What is supposed to happen with the hyperlink format you provided?

#1) I tried opening the new test case, selecting "Actions>Select Test Cases" to have the Test Run Planning window open at right. I then went to the Navigation pane and opened my Live Document with the tests to run. The Live Document opens with my test run planning in a window to the right. From here I went to the Live Document history, selected the revision I wanted to retest, switched to "Tree View" and added the tests to the run. They correctly appear as "Waiting."

If I go back to the test run and hit "Execute Tests" it populates the listing with the tests I selected, but they appear at the current version instead of the fixed revision version I wanted.
If I instead execute each test from the live document directly, while still viewing the fixed revision, the tests will be run at this version, the steps are recorded correctly at this revision, and the issue records reference the fixed revision test case. This is what I need! Thank you!
----
#2) I tried replacing the following in your suggestion:
"test_03" with my project
"testrun1" with the new "Manual" selection test run name
"00_Documents" with the wiki space that holds the Live Document
"My_Testplan_Document" with the wiki page name of the Live Document with the test step work items
"300375" with my revision number (801394)

I then pasted the link with our server appended before "/polarion" into a browser window. The response was that this wiki page did not exist yet, and it asked if I wanted to create it. I did not, so this did not lead to anything.

Code: Select all

{Our_Server}/polarion/#/project/{Project_Name}/wiki/{Space_Name}/{Page_Name}?query=(*:*%20AND%20NOT%20type:heading%20AND%20TEST_RECORDS:(%22{Project_Name}/{Test_Run_Name}%22,*)%20AND%20NOT%20TEST_RECORDS:(%22{Project_Name}/{Test_Run_Name}%22,@any))&tree_depth=0&tab=tree&revision={My_Revision}

----
#3) I tried creating a test run template from my Live Document with the query set to:

Code: Select all

(*:* AND NOT type:heading)&tree_depth=0&tab=tree&revision=801394

The part after the parenthesis is not a valid query syntax, but I figured I would try anyway. As expected, no results were returned.

Sandro_F
Posts: 19
Joined: Wed Jan 19, 2011 8:40 am

Re: Time Machine Test Runs

Postby Sandro_F » Wed May 28, 2014 1:59 pm

Well, you cannot add a revisioned test case to a test run but you can execute a revisioned test case which was added (with HEAD revision) to a test run.
The problem is that the native "Execute Tests" macro does not care about the revision and did not open the document in tree view.

You need to "hack" the "Execute Tests" macro and display it manually to be able to change target URL:

Code: Select all

             <div class="tests_execution_button" id="polarion_wiki macro name=execute-testrun-button" style="float:left">
                <div style="margin-top:0px;">
                 <form>
                  <input type="hidden" value="YOUR_URL_TO_TREE_WITH_REVISION" id="url" class="url" />
                  <input type="hidden" value="$testRun.id" id="testrun" />
                  <input type="hidden" value="Execute waiting Test Cases" id="overview" />
                  <input type="hidden" value="true" id="buttonEnabled" />
                 </form>
               </div>
             </div>

So if you have more than one document with test cases for one test run you have to find a way to keep track of these documents and display for each document an own "Execute Tests" button.


Could you please double check if the link to you live document is valid? Basically the error that the wiki-page did not exist occurs when the link to the live document is wrong.
So {Page_Name} is the name of your live document.

Regards

kaitlyn.opperman
Posts: 8
Joined: Mon Apr 28, 2014 7:08 pm

Re: Time Machine Test Runs

Postby kaitlyn.opperman » Fri May 30, 2014 8:13 pm

Sandr_F, your custom button works great. Where might I find a similar detail for the "Waiting" button? It would be nice to let users see the version of the tests that are queued to execute.
(div class="tests_waiting_button" id="polarion_wiki macro name=waiting-testrun-button" did not work)

You were correct. I had a user error with the link substitution, I had swapped two digits so that the revision I was asking for predated the wiki Live Document. With the correct version, the link works fine.

One thing to note for others. Using the original document query for my test run actually returned no results. I had to remove the "TEST_RECORDS" part of the filter before they would show. I suspect I may have another user error.

Sandro_F
Posts: 19
Joined: Wed Jan 19, 2011 8:40 am

Re: Time Machine Test Runs

Postby Sandro_F » Mon Jun 02, 2014 9:51 am

Well, you just need to adjust the query.

This query shows only test cases which are not already executed:

Code: Select all

(*:*%20AND%20NOT%20type:heading%20AND%20TEST_RECORDS:(%22test_03/testrun1%22,*)%20AND%20NOT%20TEST_RECORDS:(%22test_03/testrun1%22,@any))


This query shows all test cases:

Code: Select all

(*:*%20AND%20NOT%20type:heading%20AND%20TEST_RECORDS:("%22test_03/testrun1%22"))


So you can provide two "execute" buttons. I our case we just put the two buttons in a table with heading "All" and "Not Executed".

Furthermore you can also display the count of test cases for each button. Just execute the query over API call "queryWorkItems(...)" and call "size()" method of returned work item list.

Regards,
Sandro Frenzel (TechniSat Digital)


Return to “Polarion Application Lifecycle Management (ALM)”

Who is online

Users browsing this forum: No registered users and 4 guests