Importing private repos from Github

One approach for your projects is create libs and share it for each application. This reduce the time spent to create a new project and makes more easy the work of your team.

Golang treat each library as a repository and you can "host" them directly on Github. Ok, this mean that I can create a custom code, push it and share to my projects, but how can I do this with a private repository?

Let's go!

Using go get

To download a library/module you need to use the command go get . This command basically clone the repository on your machine, using GOPATH and GOMODCACHE locations (to see this locations use the command go env [var ...] ).

When you work with a private repository you must use some kind of authentication. Usually is used a ssh key to communicate with Github, but go get uses the clone over https, this mean that you need to use your credentials or a personal token.

There's some explanation about this in the Golang's FAQ. You will see the same link on your terminal if you try to do the go get in a private repo, for example:

$ go get github.com/rthier/private-repo
go get: module github.com/rthier/private-repo: git ls-remote -q origin in ... : exit status 128:
 fatal: could not read Username for ‘https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

Using https you will have an issue: to insert your credentials your console/terminal must be able to ask you for them and this is totally unproductive and unsafe if you use some automated process or script to get the project dependencies, like your pipeline to build the application.

You have 2 ways:
1. Use a personal access token, or
2. Use ssh instead of https for the repository (my recommended approach)

SSH instead HTTPS

This procedure is very easy, you just need to execute this on your terminal:

git config --global url."ssh://git@github.com".insteadOf "https://github.com"

Check on your .gitconfig file if the configuration was inserted:

$ cat ~/.gitconfig
# This is Git’s per-user configuration file.
[url “ssh://git@github.com/”]
        insteadOf = https://github.com/

Lets try go get again!

You will receive an error 410 Gone like the example below:

$ go get github.com/rthier/private-repo
github.com/rthier/private-repo@v0.1.0: verifying module: github.com/rthier/private-repo@v0.1.0: reading https://sum.golang.org/lookup/github.com/rthier/private-repo@v0.1.0: 410 Gone
 server response:
 not found: github.com/rthier/private-repo@v0.1.0: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/...: exit status 128:
  fatal: could not read Username for 'https://github.com': terminal prompts disabled
 Confirm the import path was entered correctly.
 If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

Well, this is expected because Golang has a mirror repository with all public modules, providing a checksum for safety controls.

UNDERSTANDING go get: when you submit a go get <module/repo> command you will not just clone the repository from Github but check the module in a mirror provided from Golang server, if the mirror doesn't contain the desired module/repo it will try to clone from the original repository saving in the Module Mirror creating a checksum.

Golang server can't reach your private repository and for this reason you need to tell to your local environment "My module is private, please don't try to mirror it and get directly from my repository" setting the environment GOPRIVATE

$ go env -w GOPRIVATE="github.com/<my_user>/<my_privaterepo>"

I strongly recommend to set GOPRIVATE with your repository and not just with your user/org if you are using the public domain and not an enterprise version of Github, this prevent some unexpected behaviors.

# DON'T DO THIS!!!!
$ go env -w GOPRIVATE="github.com/<my_user>"

More information about Golang Module Mirror and GOPRIVATE:
- https://sum.golang.org/
- https://go.dev/ref/mod#checksum-database
- https://go.dev/ref/mod#private-modules
- https://pkg.go.dev/cmd/go#hdr-Configuration_for_downloading_non_public_code

Conclusion

Using private repositories you can increase the possibilites to reuse codes creating libraries and sharing them in your projects. But keep in mind that Golang always try to clone your repo/module to a Module Mirror creating a checksum, for this reason you need explicitly set the Go environment to ignore this behavior when you go get your private repo/module.