Native Jira Integration: Technical Overview

Integration service access security

We use OAuth2 for securing our internal APIs, each service call must contain an OAuth token with a special set of scopes requested by the service.
Data isolation
Native integration use Postgres as underlying storage. Each Postgres table has an account column. This account column is used to split one account data from another during the operating. A set of integration tests verify that data is isolated on the regular basis.
Note: not actual for Private Cloud / On-premises instances.

List of Jira APIs used by integration 

GET rest/agile/1.0/board 
GET rest/agile/1.0/board/{boardId}/project 
GET rest/agile/1.0/board/{boardId}/sprint
POST rest/agile/1.0/sprint
GET rest/agile/1.0/sprint/{sprintId} 
GET/POST rest/agile/1.0/sprint/{sprintId}/issue 
POST rest/agile/1.0/epic/none/issue 
GET rest/agile/1.0/epic/{epicKey} 
POST rest/agile/1.0/epic/{epicKey}/issue 
POST rest/api/2/issue 
POST rest/api/2/issue/bulk 
GET/PUT rest/api/2/issue/{issueKey} 
GET/POST rest/api/2/issue/{issueKey}/comment 
GET rest/api/2/issue/{issueKey}/comment/{commentId} 
POST rest/api/2/issue/{issueKey}/attachments 
GET/POST rest/api/2/issue/{issueKey}/transitions 
GET rest/api/2/issue/createmeta 
GET rest/api/2/attachment/{attachmentId} 
GET rest/api/2/issuetype
GET rest/api/2/issuetype/{issueTypeId} 
POST /api/2/issueLink 
DELETE /api/2/issueLink/{issueLinkId} 
GET rest/api/2/issueLinkType 
GET rest/api/2/project 
GET rest/api/2/project/{projectKey}/statuses 
POST rest/api/2/search 
GET rest/api/2/field
GET rest/projectconfig/1/workflowscheme/{projectKey} 
GET rest/projectconfig/1/workflow 
GET rest/auth/latest/session

Additional Jira Cloud API only: 

GET rest/api/2/project/{projectId}/hierarchy 

Additional Jira Server API only: 

GET /rest/api/2/issue/createmeta/{porjectId}/issuetypes 
GET /rest/api/2/issue/createmeta/{porjectId}/issuetypes/{issueTypeId} 

Headers sent by Apptio Targetprocess

"host",
"connection",
"content-length",
"client-ip",
"content-type",
"accept",
"authorization",
"user-agent",
"accept-encoding",
"traceparent",
"tracestate",
"elastic-apm-traceparent"

Authentication

Jira Data Center and Jira Server

1. Basic authentication: User name / Password
   1.a Basic (Login/Password) + Certificate**
2. OAuth 1.0
3. Certificate (based auth. TLS Client Authentication)**
4. Gateway API Key
   4.a Gateway API Key + Certificate**
5. OAuth 2.0 Client Credentials
   //This covers use case of authentication at a gateway level with OAuth2.0 and then gateway authorizes direct connection to Jira via Basic, OAuth 1.0 or certificate. That is not for direct connection from ATP integration to Jira via OAuth 2.0. Some Jira APIs, like REST API for syncing sprints and requesting Jira boards, doesn't support OAuth 2.0 (Jira ticket: https://ecosystem.atlassian.net/browse/ACJIRA-1815; https://developer.atlassian.com/server/jira/platform/rest-apis/#authentication-and-authorization)
6. Personal Access Token
   6.a Personal Access Token + Certificate**

**Note, that accounts hosted on public clusters, cannot use self-signed certificates, only valid SSL Certificate signed by Trusted CA will work. Authorization with self-signed certificates can be implemented for private cloud customers only.

Jira Cloud

1. Basic: Email/User API Token
2. OAuth 1.0

Data storage format

Database

We use Postgres DB as the main storage for integration.

What is stored in Postgres:

• Integration profile
• Credentials. Credentials are encrypted with AES-256 algorithm (see details below in paragraph 5)
• Routings. Targetprocess and Jira projects (their IDs and names)
• Types mappings, types ids and names, types fields and fields names
• Entity shares. Pair of Targetprocess entity and linked issue in Jira. For entities we do not store any data except their issue key / entity ID and their entity type.

Database hosts on the same node inside the Kuberntes cluster.

Caching

We use in-memory cache for some almost static information(up to 30min):

• Jira instance types
• Jira instance fields
• Jira instance create meta api response
• Configured profiles info

Logs

We use Elastic Cloud for logging, no sensitive data is stored in the logs. Elastic Cloud in Ireland is used and logging can be disabled per request. For on-premise installations there is an option to use separate dedicated elastic instance.

RabbitMQ

We use rabbit MQ for underlying service communication. Rabbit messages contain changes done in one tool and changes that need to be applied to the target tool.

Rabbit is located on the same Kubernetes nodes that Jira Integration service is.

Credentials encryption

All credentials are stored in the PostgreSQL database along with other integration data in an encrypted form with AES-256 algorithm. Overview diagram and detailed flow below:

• Public/private key pair generated during creation of the cluster, private key stored locally. ( using PKCS#7 with RSA 2048 bit)
• There's "secrets" git repository created per cluster that contains encrypted or non-encrypted secrets depending on the sensitivity of the data.
• Jira integration tokens encryption key(to be used by AES256 algorithm) is automatically(encryptor service) or manually (DevOps Team, that's our case) generated and encrypted with public part of the cluster key.
• Jiraintegration service invokes Secrets / Encryptor services and send "Data Encryption key"
• Data Encryption key is decrypted using Private part of the cluster key, returned and used to encrypt Jira integration token.
• Same process for key retrieval - Jira integration services invokes Secrets / Encryptor services to retrieve Jira integration token decryption key and caches for the duration of the container lifecycle.

Processing flow

Generic for all native integrations, every tool integration uses its own adaptor

From Jira to Apptio Targetprocess

Jira adapter service listens for webhooks from Jira account. When an update occurs in Jira, Jira adapter converts it to generic format. It can queue some additional data via api if it is needed for conversion. When an event is converted it sends it to the generic events queue.

Entity sync service listens for generic events. When a generic event is received, if it has some rules configured that match that event(for example issue updated in Jira, and this issue is already shared with some Targetprocess issue) entity sync applies configured rules and generates 1 or many commands that need to be applied to the target entity. Entity sync publishes these commands to specific adapter commands RabbitMQ queue. During mappings apply entity sync could go to specific tools via adapters api to proceed some conversions like (attachment, comment, user, state etc.)

Targetprocess adapter - receives commands that need to be applied to the target entity and apply changes via tool api. If a command contains multiple field updates but not all of them are valid, adapter split initial update to few small updates and applies all valid updates ignoring not valid.

From Targetproces to Jira

The flow from Targetprocess to Jira mirrors the Jira to Targetprocess flow described in the above paragraph. The updates are generated on the Targetprocess side and applied to Jira by the same rules.

Entity share flow

The flows are also similar, not depending on the tool. Let's take a look at the share flow from Targetprocess to Jira.

When you try share some entity to Jira:

• Entity sync checks via Targetprocess adapter API if entity matches configured route. It is in the proper project, assigned to the proper team, matches dsl etc. If it, it resolves entity share target destination(target project)
• Entity sync checks if profile has a type mapping for source entity and resolves target entity type.
• If target scope(project) and type resolved, adapter creates a target entity stub in Jiral. It tries to fill all required fields with some stubs.
• If target entity created, entity sync set up sync rules between source entity and target entity.
• Entity sync transfers Targetprocess entity state. (Values of all fields configured in profile)
• Entity sync applies configured mappings to calculate target entity state and send it as 1 or few update commands to Jira adapter.
• Jira adapter applies commands to Jira entity making it be as expected to the configured mappings.

Network connectivity overview:

There are two sets of ACL's and Security Groups used to limit access to Targetprocess instances from the Internet:

• For application - inbound to only allow HTTPS access to proxy which then routes data to corresponding server or service such as Jira integration
• For management - access to Bastion host for management allowed only from Targetprocess office or only if connected to the office via VPN.

Rate limiting, retry policy and dedicated user / endpoints

For highly loaded Jira instances with large amount of integrations, a rate limiting will be set. Native integration does not limit its requests frequency, but repeats failed requests if Jira starts to limit the requests rate.

Default Targetprocess integration retry strategy works for requests at two levels – http client requests and integration messages. Failed requests will be retried 3 times 2, 4 and 8 seconds after rejection, +/- 10 concurrent requests.

In such cases, we recommend customer provide a dedicated endpoint for accessing Jira API. Otherwise, if several separate integrations use the same endpoint, they share the rate limit and may block each other.