Sunday, July 26, 2015

Adding ssh support to the Visual Studio 2015 Git plug-in

Some background


Visual Studio 2015 includes a Git plug-in that allows to version control software using Git. The plug-in allows to publish/clone from a shared Git repository, but is currently limited to repositories that support the http/https protocol.

Many users are requesting support for the ssh protocol, but the requirement is still under evaluation:
http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/3801342-add-support-for-ssh-keys-as-alternate-authenticati

Anyhow the plug-in is based on libgit2, which optionally supports the ssh protocol. The next section describes how to rebuild libgit2 in order to enable ssh support.

Rebuilding the Git plug-in with ssh support

Visual Studio 2015 includes the source code used for compiling libgit2. It is compressed in the zip file:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\4ygda0r0.oxa\libgit2-src.zip

The folder 4ygda0r0.oxa in the above path could change from installation to installation; anyhow it is enough searching for libgit2-src.zip in:
C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions

To build the source code it is necessary to:
  1. download cmake binaries, available at http://www.cmake.org/
  2. download the python interpreter, required by cmake and available at https://www.python.org
  3. download libssh2 source, available at http://www.libssh2.org/
  4. have installed Visual Studio 2015 with support for native C++ development
I think that any recent version of cmake and libssh2 should work. Anyhow, for building my own libgit2, I used the following versions:
  1. cmake 3.3.0 (while the link points to the cmake installer, the web site distributes also a zip version of the cmake binaries. In case you prefer to download this version, it is enough to unzip it and add the bin folder to the PATH)
  2. python 2.7.10
  3. libssh2 1.6.0
After installing cmake and python, unzip both the libgit2 source and the libssh2 source in two separate folders. Lets assume they are respectively c:\dev\libgit2 and c:\dev\libssh2.

To configure libgit2 for the build, open a Visual Studio 2015 command prompt configured for native 32 bit builds (from the Start Menu: Visuals Studio 2015 -> VS2015 x86 Native Tools Command Prompt) and execute the following commands:

cd c:\dev\libgit2
mkdir build
cd build
cmake -DEMBED_SSH_PATH=C:/dev/libssh2 -DSTDCALL=ON ..

The cmake command options are required in order to:
  1. the first option implies that we want to build using libssh2 and provides the path of the library. It is important to specify the path using the slash instead of the back slash;
  2. the second option specifies the calling convention, that must be stdcall in order to generate a library that interacts properly with Visual Studio
After executing the cmake command, the build folder contains a Visual Studio 2015 solution and some projects. Open the solution, change the configuration from Debug to Release and build the git2 project.

Visual Studio will generate a Release folder under build, containing git2.dll. In order to "install" the new library execute the following steps:
  1. close Visual Studio so that the git2 library is not in use and locked
  2. go into the folder: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\4ygda0r0.oxa
  3. Rename the file git2-msvstfs.dll to keep a copy of the original library
  4. copy the just built git2.dll into the folder, naming it git2-msvstfs.dll.
After executing the above steps, the new libgit2 with ssh support is now in place. Start Visual Studio 2015 and configure a remote repository using the ssh protocol. It is important to specify both the username and password in the ssh URL as in this example:

ssh://user:password@githost/path/to/repository

This is necessary because Visual Studio 2015, when calling the Git library, does not provide a callback to retrieve username and password. The libgit2 library could be modified to open a standard Windows dialog to collect this information if it is not part of the URL, but possibly this will be the subject of a future post.

Visual Studio 2015 is now able to interact with your ssh Git repository.

34 comments:

  1. Hi Bernardo,

    I have tried the above instructions and run into a few issues.

    1, for me libgit2 is located in 'C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\3bkjwnxj.mwh' not 4ygda0r0.oxa as described.

    2, python is also a dependency for cmake the create the solution.

    3, after finally getting the the solution to build and replacing the dll as described. VS is still telling me "The SSH protocol is not currently supported."?

    I had hoped to get this working then dig in and see if i could take it further and get it working with my sshkeys but would like to get the basics working first. (I also might need to pick your brains to get that working :) )

    Cheers

    Carl

    ReplyDelete
    Replies
    1. Hi Carl, I updated the article with your suggestions. Have you been able to make the recompiled library work with the ssh protocol?

      Delete
  2. If I remember correctly, yes, python is needed. But currently I have no access to the system I used for compiling the library. I'll check tomorrow and let you know.
    About the error "The SSH protocol is not currently supported.", it is possible that for some reason libgit2 was not able to find libssh2 and thus decided to compile without it. Can you check if the libgit2 project generayed by cmake contains also the libssh2 sources?
    I can also provide you the size of the compiled library so that you can check if it matches with the size of your library.

    Bernardo

    ReplyDelete
  3. I had to change:
    cmake -DEMBED_SSH_PATH=C:/p4/libssh2 -DSTDCALL=ON ..
    into
    cmake -DEMBED_SSH_PATH=C:/dev/libssh2 -DSTDCALL=ON ..
    to be able to compile.
    But still no luck. "The SSH protocol is not currently supported."
    And, yes, the VC++-project file git2 does include all references to C:\dev\libssh2.
    My previous git2-msvstfs.dll was 699kB and the newly built is 840 kB, so it seems that it was built in.

    ReplyDelete
  4. Does it works for links like git@hostname:username/repo/path.git? I'm getting "Failed to start SSH session: Unable to exchange encryption keys" error

    ReplyDelete
    Replies
    1. This type of URL does not work, because they include no password (only the username). In this case, probably, the libssh2 tries to find a suitable set of keys, but fails in locating them.

      Delete
  5. This comment has been removed by the author.

    ReplyDelete
  6. Same issue of Mattias Vannergård. All is done great without error, new git2.dll is 840Ko and i have always "The SSH protocol is not currently supported." message.
    Max

    ReplyDelete
    Replies
    1. I don't know how or why but now it's works nice...
      Thanks for this topic.

      Max

      Delete
    2. The message about the ssh protocol usually appears when entering an ssh URL in the synchronization dialog. In that case Visual Studio appears to refuse the ssh protocol.

      But if the ssh repository is added to the git configuration first and then used to synch, no problem occurs.

      Delete
    3. Which window is the git configuration on?

      I tried to clone from the team explorer connect windows; in the Local Git Repositories section and am still getting "The SSH protocol is not currently supported" error. I have the new git2.dll built and installed (file size 840K).

      Delete
    4. Which window is used to add the git configuration?

      I tried to clone my remote repository on Team explorer connect window, under local git repositories, and am still getting "The SSH protocol is not supported" error.

      Delete
    5. I think Visual Studio has some blocks preventing usage of ssh urls. To make this work do as follows:
      1) enter an http url pointing to a fake hostname
      2) publish to this invalid repository: the publish will fail, because the hostname is unreachable, but the url is anyhow saved in the git settings
      3) now go to the git settings and replace the fake http url with the real ssh url
      4) publish and now the ssh url works fine

      On the other side if you try using it from the beginning, VS will complain that the ssh protocol does not work.

      I will update the article with the exact steps and some screen captures.

      Hopes it helps.

      Delete
  7. Could you post the git2.dll file instead?

    ReplyDelete
  8. Hi Bernardo, thanks for your post.

    I could generate correctly the git2.dll, but in my case the connection is not working due to the following behavior/configuration in our Git:

    - Every time that I try to connect, I get the following error: "authentication required but no callback set".

    - Our Git repository is set with ssh rsa-key, so my Admin says that is "impossible to know the original password as is codified in MD5 and the login only is possible using rsa key pairs".

    Do you know if there is any possibility to fix this lack regarding ssh rsa-keys?

    Many thanks for your time and effort!

    Regards,

    Leandro.

    ReplyDelete
    Replies
    1. While the library would support key based authentication, there is currently no way to specify the key location. I was thinking about developing a patch that would allow to prompt users to either enter the password or specify the key in a dialog that appears when connecting to GIT.
      I had no time to work on this, but if I'll be able to develop such a patch I'll share it.

      Delete
  9. Just created a simple patch in libgit2/src/transport/ssh.c behind the block
    else if (t->owner->cred_acquire_cb) { ...}
    else if (user) {
    const char *val = NULL;
    val = getenv("HOME");
    if (val)
    {
    char *szprivfilename = malloc(strlen(val) + 128);
    char *szpubfilename = malloc(strlen(val) + 128);

    strcpy(szprivfilename, val);
    strcat(szprivfilename, "/.ssh/id_rsa");
    strcpy(szpubfilename, val);
    strcat(szpubfilename, "/.ssh/id_rsa.pub");

    git_cred_ssh_key_new(&t->cred, user, szpubfilename, szprivfilename, "");

    free(szprivfilename);
    free(szpubfilename);
    }
    if (!t->cred) {
    giterr_set(GITERR_SSH, "git_cred_ssh_key_new failed to initialize SSH credentials");
    goto on_error;
    }
    }

    ReplyDelete
  10. I did as you documented above but am getting the following.

    :\dev\libgit2\build>cmake -DEMBED_SSH_PATH=C:/dev/libssh2 -DSTDCALL=ON ..
    -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
    -- Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)
    -- zlib was not found; using bundled 3rd-party sources.
    -- LIBSSH2 not found. Set CMAKE_PREFIX_PATH if it is installed outside of the default search path.
    -- Configuring done
    -- Generating done
    -- Build files have been written to: C:/dev/libgit2/build

    I was able to build the libgit dll but without ssh support. Am I supposed to build libssh first?

    Any help would be appreciated

    ReplyDelete
    Replies
    1. No, it is not necessary to build libssh before building libgit2. The source files of libssh are included in the libgit2 project, and compiled while compiling libgit2.

      Be sure that the C:/dev/libssh2 path contains the source for libssh2; the libssh2 folder must contain the libssh2 files and folders, without any additional intermediate directory.

      Delete
    2. It is there....

      C:\dev\libgit2\build>dir ..\..\libssh2\
      Volume in drive C has no label.
      Volume Serial Number is 4800-A015

      Directory of C:\dev\libssh2

      03/04/2016 03:40 PM DIR .
      03/04/2016 03:40 PM DIR ..
      09/23/2009 03:40 PM 11,037 acinclude.m4
      02/14/2016 05:55 PM 44,090 aclocal.m4
      03/04/2016 02:25 PM DIR bin
      03/03/2010 05:00 PM 607 buildconf
      02/23/2016 02:56 AM 9 ChangeLog
      03/04/2016 02:22 PM DIR cmake
      02/23/2016 02:50 AM 3,469 CMakeLists.txt
      09/17/2013 04:36 PM 7,333 compile
      06/04/2012 01:47 PM 44,826 config.guess
      06/11/2012 07:51 AM 18,163 config.rpath
      06/04/2012 01:47 PM 35,454 config.sub
      02/14/2016 05:55 PM 599,999 configure
      02/14/2016 05:25 AM 12,082 configure.ac
      08/30/2014 05:41 PM 1,886 COPYING
      06/04/2012 01:47 PM 15,936 depcomp
      03/04/2016 02:22 PM DIR docs
      03/04/2016 02:22 PM DIR example
      08/17/2010 04:03 PM 750 get_ver.awk
      03/04/2016 02:22 PM DIR include
      06/04/2012 01:47 PM 9,233 install-sh
      01/17/2016 11:10 AM 491 libssh2.pc.in
      08/06/2015 05:32 PM 283,672 ltmain.sh
      03/04/2016 02:22 PM DIR m4
      01/18/2016 07:41 AM 4,521 Makefile.am
      02/23/2016 02:56 AM 34,245 Makefile.in
      03/15/2014 02:25 PM 351 Makefile.inc
      03/15/2014 02:25 PM 60 Makefile.libgcrypt.inc
      03/15/2014 02:25 PM 56 Makefile.OpenSSL.inc
      01/18/2016 07:41 AM 58 Makefile.os400qc3.inc
      03/16/2014 12:59 PM 54 Makefile.WinCNG.inc
      04/05/2011 12:15 PM 2,261 maketgz
      06/04/2012 01:47 PM 11,014 missing
      02/23/2016 02:56 AM 183,411 NEWS
      12/04/2014 04:43 PM 675 NMakefile
      03/04/2016 02:22 PM DIR nw
      03/04/2016 02:22 PM DIR os400
      03/12/2015 06:16 PM 474 README
      02/23/2016 02:24 AM 2,492 RELEASE-NOTES
      03/04/2016 02:22 PM DIR src
      09/17/2013 04:36 PM 3,977 test-driver
      03/04/2016 02:22 PM DIR tests
      03/04/2016 02:22 PM DIR vms
      03/04/2016 02:22 PM DIR win32
      31 File(s) 1,332,686 bytes
      14 Dir(s) 13,277,134,848 bytes free

      Delete
    3. Any resolution on this? I'm getting exact same error.
      Thanks for any help in advance.

      Delete
    4. I have the same issue as well...

      Delete
    5. Something is wrong in CMakeLists.txt from libgit2 sources... Find the lines:

      IF (USE_SSH)
      PKG_CHECK_MODULES(LIBSSH2 libssh2)
      ENDIF()

      And replace them by

      IF (USE_SSH)
      FIND_PACKAGE(LIBSSH2)
      ENDIF()

      That should do the work :).

      Delete
  11. Thanks you Bernardo for this excellent post and and "btrappe" for your comment.
    I managed to have Visual Studio 2015 being able to connect to my repository running on my Synology Disk Station running DSM 6.0 RC.

    I ran into several issues:
    1. Synology SSH server is not configured by default (security level set to High) with any cipher algorithm that libssh is accepting.
    Synology supports: aes128-ctr, aes128-gcm@openssh.com, aes192-ctr, aes256-ctr, aes256-gcm@openssh.com, chacha20-poly1305@openssh.com
    Libssh supports: aes256-cbc, rijndael-cbc@lysator.liu.se, aes192-cbc, aes128-cbc, arcfour128, arcfour, 3des-cbc

    I first tried to define LIBSSH2_AES_CTR in wincng.h:
    Ln 58: #define LIBSSH2_AES_CTR 1
    It allows kex_agree_crypt function to find a common cipher algorithm (aes256-ctr) but generates access violation exceptions later on.

    I chose the easiest path and added aes256-cbc to Synology ssh settings. It worked better.

    2. Since I’m using keys, I ran into the next issue: "authentication required but no callback set".
    Thanks to btrappe, I’ve been able to patch ssh.c file although I had to adapt the code to fit to my version of gitlib that is probably different.
    At the beginning of function request_creds, I have now:
    if (!t->owner->cred_acquire_cb) {
    if (user) {
    const char *val = NULL;
    val = getenv("USERPROFILE");
    if (val)
    {
    char *szprivfilename = malloc(strlen(val) + 128);
    char *szpubfilename = malloc(strlen(val) + 128);

    strcpy(szprivfilename, val);
    strcat(szprivfilename, "/.ssh/id_rsa");
    strcpy(szpubfilename, val);
    strcat(szpubfilename, "/.ssh/id_rsa.pub");

    git_cred_ssh_key_new(&cred, user, szpubfilename, szprivfilename, "");

    free(szprivfilename);
    free(szpubfilename);
    }
    if (!cred) {
    giterr_set(GITERR_SSH, "git_cred_ssh_key_new failed to initialize SSH credentials");
    return -1;
    }
    }
    else
    {
    no_callback = 1;
    }
    } else {

    Now everything works fine!
    Thanks again.

    ReplyDelete
  12. You can build libgit2 w/SSH for Visual Studio Update 3 (this is what I finally got it working with).

    Build libssh2 1.7.0 (you can build using 1.6.0 but you'll need to fix a couple of compile issues with VS 2015):
    * mkdir build
    * cd build
    * cmake -DCRYPTO_BACKEND=WinCNG -DBUILD_SHARED_LIBS=OFF -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF ..
    * open the resulting build\libssh2.sln in VS2015, edit the libssh2 project and set the calling convention to __stdcall (/Gz), set the build type to "Release" and rebuild all (Note: Must be Release!)

    Build libgit2
    * mkdir build
    * cd build
    * cmake -DSTDCALL=ON -DSTATIC_CRT=OFF -DUSE_SSH=OFF -DLIBSSH2_FOUND=TRUE -DLIBSSH2_INCLUDE_DIRS=/include -DLIBSSH2_LIBRARIES=/build/src/Debug/libssh2.lib ..
    * Open the libgit2.sln solution that make made
    * Make sure to set the build type to "release"
    * If you use SSH keys, do Laurent's patch to src/transports/ssh.c above, e.g.replace the following bit in request_creds (line 444 in the libgit2 src included in Visual Studio Update 3)
    if (!t->owner->cred_acquire_cb) {
    no_callback = 1;
    } else {
    With his code block from above
    * Rebuild all, then fallow randomswdev's instructions for copying the newly build libgit2 to your VS 2015 extensions directory.

    ReplyDelete
    Replies
    1. if set the build type of libssh2 to "Release" when you build libgit2 should be DLIBSSH2_LIBRARIES=/build/src/Release/libssh2.lib ..

      Delete
    2. That's correct, the LIBSSH2_LIBRARIES path should be "release" instead of "debug".

      Delete
    3. This is the closest I've gotten to getting this to compile, but libgit2 still fails (to link?) with "unresolved external symbol" errors:

      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_session_last_error@16
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_process_startup@20
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_read_ex@16
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_write_ex@16
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_close@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_free@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_session_free@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_init@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_connect@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_list_identities@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_get_identity@12
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_userauth@12
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_disconnect@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_agent_free@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_userauth_password_ex@24
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_userauth_publickey_fromfile_ex@24
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_userauth_publickey@24
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_session_abstract@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_userauth_keyboard_interactive_ex@16
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_session_init_ex@16
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_session_startup@8
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_session_set_blocking@8
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_hostkey_hash@8
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_open_ex@28
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_channel_set_blocking@8
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_userauth_list@12
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_userauth_authenticated@4
      2>ssh.obj : error LNK2001: unresolved external symbol _libssh2_init@4

      It generats a .lib file, but not a .DLL. Any suggestions? The libssh2 seemed to compile cleanly.

      Delete
  13. Compilation worked fine, i got the git2.dll in the build/release folder and renamed/replaced it with the visual studio original one.

    When i now start visual studio and put ssh://user:pass@host:2222/myrepo.git

    then sync, the console says unsupported url protocol.

    ReplyDelete
    Replies
    1. any solution for this? I am receiving same error in visual studio?

      Delete
    2. did you find solution for your problem. i am receiving same error

      Delete
  14. I complied all. But when im compiling git2 i got git2.lib and if im trying to compile as .dll it throws a couple of errors

    ReplyDelete
  15. Compiling libgit gives me a git2.lib and no dll.
    If i try to compile as dll there much errors. Somebody can help?

    ReplyDelete
  16. This is great post, Mr.Pastorelli. I followed the step you post.
    Then I tried to connect but it returned unsupported URL protocol error. Do you have any idea why this is happening?

    ReplyDelete