Native Jira Integration: Data Validation Report

Two-way data synchronization implies that both systems can be inaccessible for different periods of time and data can get out of sync. With the data validation report, you are able to find the items that are not synced properly and fix the consequences right from the report.

Who can access the report?

Targetproccess Administrators, as they have access to Targetprocess Settings –> Integrations and therefore to ‘Data validation report’ tab in every issue-level integration profile (in first release Jira or Targetprocess integrations)

What errors could be found and fixed?

With the report, you can find three groups of errors or improper sync consequences.

1. Conflicting fields: out-of-sync fields in a pair of linked issues.
2. Lost and not imported items: issues that match the integration settings and could be imported.
3. Invalid pairs: pairs, where one of the issues does not exist or its ID/Type changed and issues were lost during sync.

Scopes of the report

The first option ‘Scan synced items for conflicts’ will scan through all pairs of issues in sync. With no additional filter all synced entities will be scanned and this can take hours.

Additional fitlers are recommended. You can use DSL filters as in the Targetprocess boards and lists and scan conflcits for entities from one project or release.

For example, it could be ?ModifyDate == '20-Jul-2023' to find if there are mismatches in the data updated on July 20, 2023 in Targetprocess.

Or it could be a JQL fitler to select items for scanning from Jira perspective: Sprint IN openSprints() or  project = "Credit Card Payments"

In case a conflicting pair is valid, fields values will be compared and non-matching fields will show the results in ‘Conflicting items’ tab:

If the pair is not valid – one of the issues does not exist or change its ID/Type and sync is not possible – it will appear in the ‘Invalid links’ tab of the report:

The next two report options are for detecting not imported issues, which are two separate requests to Jira and Targetprocess.

By default, with no filters added, the report will search across all the projects selected in ‘Projects’ tab (static scopes). If JavaScript routing is added, then all accessible projects will be scanned despite the projects specified in JavaScript (if any).

Conflicts and resolutions

1. 'Conflicting fields'

In this release, the report scans all existing pairs of linked issues and compares field values according to field mappings. ‘Conflicting items’ report tab shows all fields where values do not match when they should.

Conflicts in fields mapped with JavaScript

https://github.com/TargetProcess/JS-mapping-templates/blob/main/TargetprocessMultiselectToJiraComponents.md

Comparator for JavaScript mapping of Jira Components to Targetprocess multi-select custom field

const source = args.sourceFieldValue.toolValue 
const target = args.targetFieldValue.toolValue 

const sourceArray = Array.isArray(source) ? source : [] 
const targetArray = Array.isArray(target) ? target : [] 

return sourceArray.sort().join(',') === targetArray.map(x => x.name).sort().join(',')

Conflicts in mapped collections

Collections, relations, and parent/child hierarchies are compared by a number of issues that are not synced to the second tool.

Collections and relations do not have a history of changes, therefore ‘Last change’ strategy based on a later modification date will not be accessible nor applied to selected collection conflicts. This can be fixed with Push and Pull applied one by one in order to interchange the unsynced items.

Collections can only be fixed in one way – creating all missing issues on the other side, using Pull or Push. It is not possible to remove odd relations at this time. To add all missing relations in both tools, you need to call both methods one by one - ‘Pull to Targetprocess’ all relations from Jira, and then ‘Push to Jira’ all missing relations from Targetprocess.

2. 'Invalid links'

This report’s tab shows all pairs where one or both issues do not exist – either deleted or their ID or type was changed and cannot proceed to sync.

This category of errors has the following actions:

• 'Refresh conflicts'
• 'Unlink'

3. 'Not imported'

All issues that match the integration profile filters at the moment of the report scanning the tools.

To fix conflicts from the 'Not Imported' apply one of two actions – "Pull to Targetprocess" or "Push to Jira" to import all selected (filtered) entities to selected tool.

Filters and actions

Filters

All issues filtered out and viewable in the list are considered as a selected scope for any action. Should you need to apply an action to a specific row of the report, filter the specific row out and then apply changes to the only row left in the report.

Action: ‘Refresh conflict’

In case conflicted fields have their values changed, then ‘Refresh conflicts’ action will bring the actual values. If conflict is not actual anymore, it will disappear from the report (mappings removed, issues unlinked) or change its state to ‘Resolved’ (issue imported)

Action: ‘Ignore’ and ‘Unignore conflicts’

This action moves conflict to the ‘Ignored’ state. In this state, conflict will not be processed – not refreshed, not fixed. Before applying any conflict resolution strategy, you need to move conflict back from ‘Ignored’ state to an actual.

‘Unignore conflicts’ triggers conflict refresh in the background. If conflicts were fixed indirectly in one of the systems, then field conflict will move to a valid state – ‘Resolved’, if values match at this time, or back to ‘Conflict’, if mismatch is valid.

Conflict resolution strategies

To fix the conflicts in the mapped fields, you can choose one of three strategies:

1. Conflict resolution strategy ‘Latest change’

Out of two conflicting values, the latest one will be selected and applied to the matching filed in the other system.

2. Conflict resolution strategy ‘Push to Jira’

With this strategy, all values from Targetprocess fields will be sent to Jira.

In the ‘Not imported’ section, this will import all selected Targetprocess items to Jira.

3. Conflict resolution strategy ‘Pull to Targetprocess’

This strategy considers Jira as a source of truth and rewrites Targetprocess fields with values from Jira.

In the ‘Not imported’ section, this will import all selected Jira issues items to Targetprocess.

4. Conflict resolution strategy ‘Unlink’

Unlink is required if the entity has been synced and the link is not valid for various reasons (issue deleted or converted and its ID/Type changed and lost). In such cases, before importing an issue again as new it must be unlinked from the previous item.

As soon as the issues are unlinked, they move to the ‘Resolved’ state and no other actions are possible with resolved invalid links for now. The free issue can be imported manually to the connected tool.

Conflict resolution workflow

MIRO schemes of the process

Before any action is applied, every field’s conflict is verified and its status and fields are updated. This validation can be called manually with the ‘Refresh’ action.

Conflict statuses

Status ‘Conflict’

Fields or issues that do not match.

With items in the ‘Conflict’ state, you can:

• Verify the conflict by applying ‘Refresh’ action
• ‘Ignore’ field and no action will apply to them, even if such issue gets under the filter
• Fix filtered conflicts with one of the resolving strategies – ‘Pull to Targetprocess’, ‘Push to Jira’, or ‘Latest values’

Status ‘Processing’

Temporary state which indicates that the existence of conflict is being checked. If values are the same, and if an action selected (’Unignore’, ‘Pull to Targetprocess’, ‘Push to Jira’, or ‘Latest values’) can be applied to selected items.

‘Processing’ state will automatically change

• To ‘Resolved’, if conflict is fixed
• Back to ‘Conflict’ if selected change cannot be applied or conflict still occurs and no further actions were applied
• To ‘Update Sent’ if action is applied and new value is sent to Jira

Status ‘Resolved’

Final state. No actions will apply to resolved conflicts in current report version.

Status ‘Ignored’

No actions will apply to conflicts in status ‘Ignored’. The ‘Unignore conflicts’ action will automatically call a ‘Refresh’ and actualize the conflict.

Status ‘Update Sent’

This state indicates that updates are in the queue and will be applied in Jira shortly. You must manually refresh conflicts in order to know if the changes applied successfully. This state will not change automatically to the next, even if values were updated already.