Script to clone all Git repositories from your VSTS collection

August 23, 2016, (updated on September 26, 2017), 7 comments, Software Development

In this article I show you a PowerShell script which clones and pulls all Git repositories from your Visual Studio Team Services (VSTS) collection. I use this script to retrieve the complete code base so that I can run my code analysis tools on it.

1. Create a new PowerShell script file

In the directory where you want to clone all repositories, create a new CloneAllRepos.ps1 script file with the following contents:

# Read configuration file
Get-Content "CloneAllRepos.config" | foreach-object -begin {$h=@{}} -process { 
    $k = [regex]::split($_,'='); 
    if(($k[0].CompareTo("") -ne 0) -and ($k[0].StartsWith("[") -ne $True)) { 
        $h.Add($k[0], $k[1]) 
    } 
}
$url = $h.Get_Item("Url")
$username = $h.Get_Item("Username")
$password = $h.Get_Item("Password")

# Retrieve list of all repositories
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $username,$password)))
$headers = @{
    "Authorization" = ("Basic {0}" -f $base64AuthInfo)
    "Accept" = "application/json"
}

Add-Type -AssemblyName System.Web
$gitcred = ("{0}:{1}" -f  [System.Web.HttpUtility]::UrlEncode($username),$password)

$resp = Invoke-WebRequest -Headers $headers -Uri ("{0}/_apis/git/repositories?api-version=1.0" -f $url)
$json = convertFrom-JSON $resp.Content

# Clone or pull all repositories
$initpath = get-location
foreach ($entry in $json.value) { 
    $name = $entry.name 
    Write-Host $name

    $url = $entry.remoteUrl -replace "://", ("://{0}@" -f $gitcred)
    if(!(Test-Path -Path $name)) {
        git clone $url
    } else {
        set-location $name
        git pull
        set-location $initpath
    }
}

2. Create configuration file

Now, create a CloneAllRepos.config file alongside your ps1 script with your configuration:

[General]
Url=https://myproject.visualstudio.com/defaultcollection
Username=user@domain.com
Password=YourAccessToken

If you use two-factor authentication, you need to generate a password on the VSTS site: Go to User Settings > Security > Personal access tokens, create an access token and use it as password in the configuration file.

3. Execute script

To execute the script in the Windows or PowerShell command prompt, run the following command:

powershell -ExecutionPolicy Bypass -File ./CloneAllRepos.ps1

If you’d like to execute the script without bypassing the policy in each call, you have to enable script execution globally:

Set-ExecutionPolicy Unrestricted

Now, you can execute the script in the PowerShell command line in this way:

./CloneAllRepos.ps1
Tweet about this on TwitterShare on FacebookEmail this to someoneShare on TumblrShare on LinkedIn

Tags: , ,

7 responses to “Script to clone all Git repositories from your VSTS collection”

  1. Robert Bratton says:

    Thanks for the script. It does just what I needed.

  2. SvendK says:

    Thanks for the quick replies. Actually I cannot get the script to work.

    I made the two files as requested (but with right names, hehe).
    I can login on the URL via a web browser with the credentials given in the file, so url and credentials is correct.

    I get this error: https://i.imgur.com/hpP4jKh.png

    URL: http://172.16.120.22:8080/tfs/FlexPOS%20APS – this shows the main page – and with the _apis prefix from your script, I get some json with what seems to be all the projects. And if I put a collection name after APS/ in the URL, I get a smaller json also with project names.

  3. SvendK says:

    In our team at work, we try to keep the projects in the same folder across our computers.
    I’d like to be able to retrieve all collections and their projects and put it all in the same folder with collections as subfolders and their projects below them. Would it be possible to alter your script to first retrieve all collections and then create them as folders, and in them run your script for each collection?

    • Rico Suter says:

      I think there should be a URL to retrieve all collections and then you can run the script for each collection. I think MS has docs for the TFS API but I haven’t done this before…

  4. SvendK says:

    Thanks a lot for this. One thought: The PS-file should be called CloneAllRepos.ps1, but in the PS call it says CloneAllProjects.ps1 – shouldn’t this be the same?

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax