Sergey # Blog

Few thoughts about the current.

VSO CI and CodeCov

GitHub supports a number of services with which you can integrate your repo. I have recently come across CodeCov that provides code coverage results and has a nice integration with the GitHub. I decided to try it out and here what I had to do.

Prerequisites

I have C# project hosted in the GitHub, MsTest, VSO CI. I granted access to my GitHub repo on the https://codecov.io and got access key.

Nuget package

First I looked at the sample repo. When I scrolled through “CI section” and “Combine test and code coverage” it did not seem to be a one click integration… But the first instruction was very simple: install OpenCover nuget package. When I installed I noticed that is has documentation in the package and well… when I started reading I almost decided not to proceed. In reality you do not need it when you start. It has a lot of useful information but it is more for fine-tuning.

Powershell script

Next step in the instructions is a Powershell script that creates coverage file and uploads it to CodeCov. Here is my script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
..\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe -register:user 
  "-target:C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" 
  "-targetargs:..\a\Release\Src\DependencyCollector\Net40.Tests\Microsoft.ApplicationInsights.DependencyCollector.Net40.Tests.dll  [CUT: all assemblies]  
  ..\a\Release\Src\WindowsServer\WindowsServer.Nuget.Tests\WindowsServer.Nuget.Tests.dll /TestCaseFilter:TestCategory!=Required_4_5_1 /logger:trx" 
  "-filter:+[Microsoft.AI*]* -[*Tests]* -[*TestFramework*]* -[*]Microsoft.ApplicationInsights.Extensibility.Implementation.External*" -hideskipped:All -output:.\coverage.xml

(New-Object System.Net.WebClient).DownloadFile("https://codecov.io/bash", ".\CodecovUploader.sh")

$lastCommit = $(git rev-parse HEAD)

$branchNames = $(git branch --all --contains $lastCommit) 

$i=0
Foreach ($branchName in $branchNames)
{
    $i++
    if ($i -gt 1)
    {
        $branchName = $branchName.Trim()
      $lastCommitOnBranch = $(git rev-parse $branchName)
        if ($lastCommitOnBranch -eq  $lastCommit)
        {
          if ($branchName.StartsWith("remotes/origin/"))
            {
                $branchName = $branchName.Substring("remotes/origin/".Length)
            }

            .\CodecovUploader.sh -f coverage.xml -t $env:CODECOVACCESSKEY -X gcov -B $branchName
        }
    }
}

It took me some time to get to it :) So let’s look at it line by line:

1. Run tests

1
2
3
4
5
..\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe -register:user 
  "-target:C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" 
  "-targetargs:..\a\Release\Src\DependencyCollector\Net40.Tests\Microsoft.ApplicationInsights.DependencyCollector.Net40.Tests.dll  [CUT: all assemblies]  
  ..\a\Release\Src\WindowsServer\WindowsServer.Nuget.Tests\WindowsServer.Nuget.Tests.dll /TestCaseFilter:TestCategory!=Required_4_5_1 /logger:trx" 
  "-filter:+[Microsoft.AI*]* -[*Tests]* -[*TestFramework*]* -[*]Microsoft.ApplicationInsights.Extensibility.Implementation.External*" -hideskipped:All -output:.\coverage.xml

Here I run OpenCover.Console.exe that is unpacked to the package folder during build. You can check description of all arguments here.

I think I do not need register. I will try to remove it in the next version. I just kept it from the sample that I used as a start point.

In target and targetargs you specify how to run tests. I opened my VSO build, opened test step and copied the command from there.

In targetargs I specify list of all test assemblies. Notice how “” are placed. It is important! Opening “ should be before the argument or it will not work.

/TestCaseFilter is also an argument for vstest.console.exe. I need it for skipping some tests. If you need other vstest arguments read this

-filter:+[Microsoft.AI*]* -[*Tests]* -[*TestFramework*]* -[*]Microsoft.ApplicationInsights.Extensibility.Implementation.External* means include all types from assemblies which names start with “Microsoft.AI” except all assemblies that have “Tests” or “TestFramework” in the name (otherwise you wound measure code coverage of unit test projects). Also exclude all types from all assemblies where namespace starts with “Microsoft.ApplicationInsights.Extensibility.Implementation.External”.

You can improve this by:

  • Pass package folder as a parameter. But I was not sure how to propagate properties from MsBuild step to a Powershell step and did not want to spend much time on it.
  • OpenCover package version is hardcoded meaning that there will be problems with upgrade.
  • Path to vstest.console.exe should also be taken from the environment variable. That should be easy: read this.

2. Download the results

Second line is downloading coverage results uploader:

1
(New-Object System.Net.WebClient).DownloadFile("https://codecov.io/bash", ".\CodecovUploader.sh")

3. Upload the results

Now an interesting part starts :) When I was just uploading coverage results (by running CodecovUploader.sh) it was attaching it to a master branch while I was actually compiling my private branch. What was happening is that VSO instead of just checking out a branch was checking out a commit so branch was in the detached state (so “git rev-parse” would not work).

So what I am doing next is I take last commit of this detached branch and try to figure out on what branch (branches) that commit is the last one and then upload code coverage results on all these branches. If you know an easier way please post a comment! :)

Get last commit of detached branch: $lastCommit = $(git rev-parse HEAD)

Get all branches that have this commit:

1
$branchNames = $(git branch --all --contains $lastCommit)

Note that

  1. If you just created a local branch from a develop branch and did not make any changes you would have last commit present in your branch and in develop. On both branches that would be the last commit and code coverage would apply to both.
  2. If you have your local branch and made some changes you would have your last commit be present only in one branch.
  3. In reality this command also always returns “master detached” as a first element. We would need to skip it.

For each branch that has our commit from detached branch (except the first one which is “detached master”) check that our commit is the last commit for the branch, if yes than upload result:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$i=0
Foreach ($branchName in $branchNames)
{
    $i++
    if ($i -gt 1)
    {
        $branchName = $branchName.Trim()
        $lastCommitOnBranch = $(git rev-parse $branchName)
        if ($lastCommitOnBranch -eq  $lastCommit)
        {
          if ($branchName.StartsWith("remotes/origin/"))
            {
                $branchName = $branchName.Substring("remotes/origin/".Length)
            }
            
            .\CodecovUploader.sh -f coverage.xml -t $env:CODECOVACCESSKEY -X gcov -B $branchName
        }
    }
}

Note:

  • You have to Trim branch name; it has leading spaces and “git rev-parse” would fail without it.
  • You have to cut “remotes/origin/” from a branch name because CodeCov does not understand it.
  • $env:CODECOVACCESSKEY is your access key that you get from https://codecov.io. It is passed as a variable to the script. You can also pass it as a Powershell script argument if you like.
  • CodecovUploader.sh needs to be run from the git repo root. If you try to do that from a different folder, it will fail. Interestingly enough it opens a new console window, does some magic and immediately closes so if it failed you do not know why. I wonder if there is a good way to detect problems… Here is the list of arguments if you want to check yourself.

And Voila! Report is there! :)

Visualization

  • I added a badge to my repo main page. To get a badge open you repo info https://codecov.io/[YourRepo]. Click on the gear button -> badge. Insert markdown code to your readme.md.

You will see something like that:

On the top of your pull request you will see something like this:

… and for each changed file:

What I did not like

  • Tests are executed twice. Once when you run tests in the scope of your build and then to get code coverage. That increases build time. In my case we have 700 unit tests. Build time without code coverage is 9 minutes. With it 14. It is not a big deal with 3 people in the team. But if that would be 10 people with build time increase from 20 to 25 I would doubt if I should add it (Also tests run longer, I had to increase timeouts for some tests) .
  • I’m actually using IE … at work :) I want to have IE extension.
  • I did not find anything about reports retention policy and did not find how to delete reports.
  • Well… It was not really a one click experience since I have VSO CI
  • Troubleshooting of uploader. I hope I never need it because I would not know immediately how to do this. Also in case of CI I would not even notice that when it got broken if I do not check it for each pull request.
  • Test containers need to be excluded from the results. It would be nice to exclude them by default.

Comments