Gitlab Docker Container Registry Login Failure

For more information on Gitlab, visit gitlab.com

Symptoms:

---

Recently I decided to configure the container registry services on gitlab, to integrate our gitlab instance with docker, for auto image builds using the built-in gitlab CI on commit. I configured the container registry as per gitlab's instructions, or the instructions found here, and once complete, attempted to log in to the container registry from a docker host in my environment. The login promply failed with the following error mesage:

docker login registry.appcontainers.com
Username: rnason
Password:
Error response from daemon: Get https://registry.appcontainers.com/v2/: error parsing HTTP 403 response body: unexpected end of JSON input: ""

Checking the error logs yielded the following results:

tail /var/log/gitlab/registry/current:

2016-07-29_17:26:34.62640 time="2016-07-29T13:26:34.626149246-04:00" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.5.4 http.request.host="gitlab.appcontainers.com:5000" http.request.id=56a12c17-c4fd-2137-4815-6174f3579dc9 http.request.method=GET http.request.remoteaddr=10.0.0.8 http.request.uri="/v2/" http.request.useragent="docker/1.11.2 go/go1.5.4 git-commit/b9f10c9 kernel/3.16.0-4-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.2 \\(linux\\))" instance.id=83d3c6a6-40cd-4dac-c2fc-6154c58a21cf service=registry version=v2.4.1

tail /var/log/gitlab/gitlab-rails/production.log:

Started GET "/jwt/auth?account=rnason&client_id=docker&offline_token=true&service=container_registry" for 10.0.0.8 at 2016-07-29 17:26:34 -0400
Processing by JwtController#auth as HTML
  Parameters: {"account"=>"rnason", "client_id"=>"docker", "offline_token"=>"true", "service"=>"container_registry"}
Filter chain halted as :authenticate_project_or_user rendered or redirected
Completed 403 Forbidden in 501ms (ActiveRecord: 16.3ms)

Solution:

---

I double checked my configuration, and scoured the internet for 2 days trying any solution that I came across to no avail. No matter what I tried, the same results plagued my efforts.

Finally I attempted resetting my gitlab password, and attempting to login again. I happened to fat finger the new password, which resulted in the same exact error. I then tried the login both with the correct password as well as a hit random keys password which resulted in the same unexpected end of JSON input error. This lead me to think that the issue has something to do with the user account as opposed to the JWT Token Authentication mechanism that I originally thought was the problem.

I started combing through my user account, and noticed that when I entered the Admin Area, clicked on my user account from the users portion in the center bottom of the screen, and clicked identities, that Gitlab would throw a 500 error.

A quick search determined that gitlab uses the user identities to tie local Gitlab accounts to directory service accounts. Remembering that I had once had this service connected to an LDAP instance, I decided to take a look to see if I could figure out where that information was stored in the database.

I connected to the database, and checked the table structure, and sure enough there was an identities table, so I ran a query to see what was in the table.

\c gitlab

\dt identities

            List of relations
 Schema |    Name    | Type  |   Owner
--------+------------+-------+------------
 public | identities | table | gitdbowner
(1 row)

SELECT * FROM identities;

 id |                      extern_uid                         | provider | user_id | created_at | updated_at
----+-------------------------------------------------------+----------+---------+------------+--------------
  1 | uid=rnason,cn=users,cn=accounts,dc=appcontainers,dc=com | ldapmain |       2 |            | 2016-02-21 08:46:18.758738
(1 row)

hmmm suspect... seeing that there existed an entry that tied my now local gitlab user account to an old LDAP instance, I removed the entry from the database.

DELETE FROM identities WHERE id=1;

DELETE 1

SELECT * FROM identities;

 id | extern_uid | provider | user_id | created_at | updated_at
----+------------+----------+---------+------------+------------
(0 rows)

Back on the gitlab server just to be safe, I restarted the instance, with a gitlab-ctl restart and logged back into my Gitlab instance to check the identities tab.

Success.. no more 500 error! Now that the issue was cleared up, I re-attempted to do a docker login back on my docker host:

docker login registry.appcontainers.com
Username: rnason
Password:
Login Succeeded

ohh man.. things are looking good, one final test

docker tag appcontainers/debian registry.appcontainers.com/appcontainers/debian

docker push registry.appcontainers.com/appcontainers/debian
The push refers to a repository [registry.appcontainers.com/appcontainers/debian]
b70151107533: Pushed
latest: digest: sha256:621ba6f14234254de69a58a32a6f1c8195aed5fe4c0c41c7382187c21e2cc03a size: 528

and Qapla'... Problem solved!

References:

---

clusterfrak.com