Backend API for Jenkins Plugin Site
Go to file
R. Tyler Croy 1d34337a18
Properly clone the plugin-site frontend into a subdirectory instead of on top of our existing code
`git` checks out into the "current directory" which means we were technically
builting just the node-only (plugin site frontend) container before
2016-11-02 09:01:50 -07:00
deploy Ignore local checkouts of the frontend 2016-11-02 05:33:35 -07:00
src Add mock ScheduledExecutorService 2016-10-11 13:02:27 -04:00
.editorconfig Initial commit 2016-07-15 11:38:52 -04:00
.gitignore Build the data file and host it for the generation 2016-10-16 18:53:54 -07:00
Dockerfile Switch to Jetty for a smaller server container 2016-10-16 18:53:54 -07:00
Jenkinsfile Properly clone the plugin-site frontend into a subdirectory instead of on top of our existing code 2016-11-02 09:01:50 -07:00
LICENSE.txt Initial commit 2016-07-15 11:38:52 -04:00
Makefile Support building a single container for jenkins.io deployment 2016-11-02 03:54:47 -07:00
README.asc README clean up 2016-10-11 13:16:01 -04:00
pom.xml Add mock ScheduledExecutorService 2016-10-11 13:02:27 -04:00

README.asc

= Plugin Site API

This is the backend API for Jenkins Plugin Site frontend. It provides a REST API
to allow searching for plugins and retrieving specific plugin information.

== Architecture

The REST API calls are powered by Jersey 2.x running inside of Tomcat 8. Plugins
are stored and searched via embedded Elasticsearch.

Plugin information sources include:

* Jenkins Update Center
** Main source of plugin data
* Jenkins Stats
** Installation statistics
* Jenkins Wiki
** Scrape wiki content

Elasticsearch mappings are initialized at runtime using a temporary
directory that is cleaned up at shutdown. The data is generated via a separate
Maven task and stored on an external URL outside of the application. The data file is downloaded from
said URL and extracted at boot time. Every 12 hours the application will download the file again and check if it
contains new data. If so the application will the reindex the Elasticsearch data using the new data file.

== Run Local Plugin Site API

----
DATA_FILE_URL="http://url.to/plugins.json.gzip" mvn jetty:run
----

This will launch an embedded Jetty container accessible at `http://localhost:8080`.

== Run Docker Plugin Site API

----
docker build -t jenkinsciinfra/plugin-site-api .
docker run -p 8080:8080 -it -e DATA_FILE_URL="http://url.to/plugins.json.gzip" jenkinsciinfra/plugin-site-api
----

== Rebuild Elasticsearch data

----
mvn -P generatePluginData
----

This will generate a new file in `target/plugins.json.gzip` consisting of plugin information and installation
statistics. This file should be uploaded to DATA_FILE_URL.

== REST API Reference

WARNING: This API is still in flux as the frontend and backend are
still being developed. As such this API is likely to change in the near future.

Unless otherwise indicated, all API calls:

* Produce JSON
* Do not require authentication
* Have no parameters or request body

Possible error codes for all calls:

* 500

=== GET /categories

Retrieve categories with their labels. It will be necessary to call /labels
to get the titles

Sample Response
----
{
  "categories":
  [
    {
      "id": "languagesPlatforms",
      "title": "Platforms",
      "description": "Jenkins plugins that are designed to give added support for building, testing or deploying to specific languages or platforms.",
      "labels":
      [
        "ios", "dotnet", "android", "ruby", "scala"
      ]
    },
    ...
  ],
  "total": 6
}
----

=== GET /maintainers

Retrieve unique maintainers in the plugin data.

Sample Response
----
{
    "maintainers": [
        "Kohsuke Kawaguchi",
        "Jesse Farinacci",
        "Stephen Connolly",
        "Gregory Boissinot",
        "Oliver Gondža",
        "Oleg Nenashev",
        "Seiji Sogabe",
        "Lucie Votypkova",
        "Vojtech Juranek",
        "Nicolas De Loof",
        "Bap",
        "Praqma Josra",
        "Robert Sandell",
        "Stefan Brausch",
        .....
    ],
    "limit": 909
}
----

=== GET /labels

Retrieve available plugin labels. "title" is an optional field so it may be
missing from some labels.

Sample Response
----
{
  "labels" :
  [
    {
      "id": "ios",
      "title": "iOS development"
    },
    {
      "id": "dotnet",
      "title": "Azure and .NET"
    },
    {
      "id": "android",
      "title": "Android development"
    },
    {
      "id": "ruby",
      "title": "Ruby development"
    },
    {
      "id": "scala",
      "title": "Scala plugins"
    },
    ....
  ],
  "limit": 30
}
----

=== GET /plugin/:name

Retrieves information about a plugin

Could return 404 if plugin is not found

Sample Response
----
{
  "buildDate": "Jul 04, 2016",
  "categories": [
    "scm"
  ],
  "dependencies": [
    {
      "name": "matrix-project",
      "optional": false,
      "version": "1.6"
    },
    ...
  ],
  "maintainers": [
    {
      "id": "kohsuke",
      "name": "Kohsuke Kawaguchi",
      "email": null
    },
    ...
  ],
  "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
  "gav": "org.jenkins-ci.plugins:git:2.5.2",
  "labels": [
    "scm"
  ],
  "name": "git",
  "previousTimestamp": "2016-07-02T20:46:28.00Z",
  "previousVersion": "2.5.1",
  "releaseTimetamp": null,
  "requiredCore": "1.609.3",
  "scm": "github.com",
  "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
  "stats": {
    "installations": [
      {
        "timestamp": 1322697600000,
        "total": 8906
      },
      ...
    ],
    "installationsPercentage": [
      {
        "timestamp": 1459468800000,
        "percentage": 61.16896694248365
      },
      ...
    ],
    "installationsPerVersion": [
      {
        "version": "2.0.3",
        "total": 141
      },
      ...
    ],
    "installationsPercentagePerVersion": [
      {
        "version": "2.2.7",
        "percentage": 0
      },
      ...
    ],
    "currentInstalls": 89232,
    "trend": 2990
  },
  "title": "Jenkins Git plugin",
  "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
  "version": "2.5.2",
  "wiki": {
    "content": <HTML content>,
    "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
}
----

=== GET /plugins

Search for plugins

.Parameters
[options="header,footer"]
|=======================
|Name|Required|Description|Default Value|Possible Values|Example
|q|false|Search plugin name, title, excerpt if given|||workflow
|sort|false|Define how results are sorted|name|name,updated|name
|categories|false|Filter by categories|||scm
|labels|false|Filter by labels|||scm,ios
|maintainers|false|Filter by maintainers|||kohsuke
|core|false|Filter by required core version|||2.13
|limit|false|Specify page limit for results|50||25
|page|false|Specify page number to return. This is not zero based|1||3
|=======================

Sample Request
----
GET /plugins?q=git&sort=name&limit=3&page=1
----

Sample Response
----
{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}
----

=== GET /plugins/installed

Get top "limit" install plugins

.Parameters
[options="header,footer"]
|=======================
|Name|Required|Description|Default Value|Possible Values|Example
|limit|false|Specify limit for results|10||5
|=======================

Sample Request
----
GET /plugins/installed
----

Sample Response
----
{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}
----

=== GET /plugins/trend

Get top "limit" trending plugins

.Parameters
[options="header,footer"]
|=======================
|Name|Required|Description|Default Value|Possible Values|Example
|limit|false|Specify limit for results|10||5
|=======================

Sample Request
----
GET /plugins/trend
----

Sample Response
----
{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}
----

=== GET /plugins/updated

Get top "limit" recently updated plugins

.Parameters
[options="header,footer"]
|=======================
|Name|Required|Description|Default Value|Possible Values|Example
|limit|false|Specify limit for results|10||5
|=======================

Sample Request
----
GET /plugins/updated
----

Sample Response
----
{
  "page": 1,
  "pages": 3,
  "plugins":
  [
    {
      "buildDate": "Jul 04, 2016",
      "categories": [
        "scm"
      ],
      "dependencies": [
        {
          "name": "matrix-project",
          "optional": false,
          "version": "1.6"
        },
        ...
      ],
      "maintainers": [
        {
          "id": "kohsuke",
          "name": "Kohsuke Kawaguchi",
          "email": null
        },
        ...
      ],
      "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.",
      "gav": "org.jenkins-ci.plugins:git:2.5.2",
      "labels": [
        "scm"
      ],
      "name": "git",
      "previousTimestamp": "2016-07-02T20:46:28.00Z",
      "previousVersion": "2.5.1",
      "releaseTimetamp": null,
      "requiredCore": "1.609.3",
      "scm": "github.com",
      "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=",
      "stats": {
        "installations": [
          {
            "timestamp": 1322697600000,
            "total": 8906
          },
          ...
        ],
        "installationsPercentage": [
          {
            "timestamp": 1459468800000,
            "percentage": 61.16896694248365
          },
          ...
        ],
        "installationsPerVersion": [
          {
            "version": "2.0.3",
            "total": 141
          },
          ...
        ],
        "installationsPercentagePerVersion": [
          {
            "version": "2.2.7",
            "percentage": 0
          },
          ...
        ],
        "currentInstalls": 89232,
        "trend": 2990
      },
      "title": "Jenkins Git plugin",
      "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi",
      "version": "2.5.2",
      "wiki": {
        "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin"
      }
    },
    ...
  ],
  "limit": 3,
  "total": 7
}
----

=== GET /versions

Retrieve unique required Jenkins versions in the plugin data.

Sample Response
----
{
  "limit": 226,
  "versions": [
    "1.580.1",
    "1.625.3",
    "1.424",
    "1.609.3",
    "1.609.1",
    "1.398",
    "1.466",
    "1.480",
    "1.596.1",
    "1.532.3",
    "1.580",
    "1.509.4",
    "1.642.3",
    "1.580.3",
    "1.447",
    "1.532",
    "1.609",
    "1.509",
    "1.554.1",
    "1.480.3",
    "1.509.3",
    "1.392",
    .....
  ]
}

----