Targetprocess Performance Improvements

During the last quarter of 2019, it came to our attention that some users were experiencing performance issues. Consequently, we have been improving performance with the goal of fixing all major issues by the end of the first quarter of 2020.

This article is to show the progress we’ve made, what has been fixed, and what we are going to continue to do moving forward.

Key Performance Improvements Overview

• We started using CDN (Content Delivery Network) to deliver assets faster and in a more predictable way.
• As part of our microservices movement, we deliver new functionality as “UI integrations”. It allows developers to deliver bug fixes and new functionality much faster. Our initial implementation of these integrations lacked proper browser caching, and clients had to download them much more frequently than they really needed to. With our new approach, UI integrations only have to be re-downloaded when we release new versions.
• We added in-memory caches for rarely modified data, like custom field settings, processes, terms, and entity types, which allows us to serve them 99% of the time from the cache without any database queries.
• We fixed a major redundancy that was very inefficient when reviewing business rules on the Assignable’s entity states, which used to cause additional queries on large accounts.
• We revisited our typical patterns of database queries and were able to make them much faster for "hot" scenarios, like building Live Notifications or grouping cards by Lanes

Each line represents an average time it takes to load an app, in seconds (for different production clusters)

How You Can Benefit From Performance Improvements

If you update Targetprocess regularly: Since performance is the prioritized objective for our team, we have been improving the application performance throughout the current quarter and will keep working on it in the next quarter. Each new build we deploy includes the latest updates we have implemented and will slightly enhance the performance of your application. There’s no need to make any special changes, just make sure you are using the latest version. Please note, in this instance, you probably wouldn’t notice a drastic change in performance at first glance.

If you don’t update Targetprocess regularly: Please contact your Account Manager in order to update up to the latest build of the Targetprocess application. If you haven’t updated your system for several months, you would definitely see significant changes in the application’s performance. One of our large clients saw 30 times faster Context API performance, 1.75 times faster initial load time, and 4.5 times faster automation rules processing after upgrading to the latest version.

Performance Improvements with Technical Details

CDN

Our frontend application is quite large and has to download several megabytes from various assets (JavaScript, css, fonts) upon launch. Using CDN allows us to provide a more consistent and predictable connection, serve assets using HTTP/2 with no additional configuration, and serve assets from a separate host, which avoids the browser limit of concurrent HTTP requests per host.

We still support no-cdn mode for isolated on-premises installs and some on-demand accounts with connection issues to Cloudflare.

HTTP updates

Caching of assets

We added proper browser caching for the mentioned frontend assets, both with and without CDN. As a result, these large assets are cached for 1 year the first time a user opens Targetprocess, and are not fetched again from the server until a new build is deployed.

Caching of UI integrations

All new features are now delivered as UI integrations, external to the frontend. This allows developers to deliver bug fixes and new functionality much faster. This new approach allows the browser to cache integration for a long time, only updating when a new version is released, which means the app spends almost no time downloading these files.

HTTP caches and Board.aspx

Similar to the improvements to assets, we also added version-based caching support to many Targetprocess API endpoints. As a result, the following APIs are cached most of the time, and the frontend doesn’t make any calls:

• Mashups bundle
• Feature-toggle
• Extendable domain schema
• Native domain schema
• Entity types
• Localized fields
• Meta for visual filters

Redundant Auth requests

While only minor, this caused significant delays. We stopped the frontend from making unnecessary calls to Auth Service to retrieve its meta-information, instead performing initial authentication by injecting this data directly into Board.aspx. We also fixed a bug in openid-client.

Customized layout optimizations

We performed some optimizations to the Layout Service usage, so opening cards with a customized detailed view should make less HTTP requests.

View menu optimizations

We’ve made big changes to the backend for Views API and reduced the amount of CPU and RAM consumption by 5 to 10 times. Views API is now up to 8 times faster for large clients.

Prometheus metrics

We’ve added the AppMetrics package to Targetprocess, created the infrastructure to collect metrics periodically and relay them in the Prometheus format.

Analyzing comet performance

We’ve optimized hot paths of data querying, like expression parsing, expression compilation, SQL filters etc. It’s already better than it was at the end of last year, especially on DAL cluster, but on AMS we still see some issues, but we hope to resolve them soon.

Resource conversion and validation

The application spent too much time doing unnecessary work, such as deserializing and validating custom fields when it didn’t make sense to. Our improvements have allowed us to significantly increase performance of API v1 modifications.

API v2 parser caching and optimizations

We made some improvements to avoid caching incorrect expressions and re-implemented many expression visitors to be more efficient. About 99.5% of parsed expressions are now taken from the cache.

In-memory server caches

Targetprocess often needs data which is rarely modified, like Custom Field definitions, Terms, Entity Types, Practices, Extendable Domain schema etc. We added caches and introduced ‘component versions’, which quickly allows it to find out if custom fields match the database state.

For example, we saved 60-80% of the time required for Targetprocess to take custom fields from the in-memory cache. The rest of the time it fetches a single version number from the database (which is executed almost immediately) or the database state of custom fields because they were modified.

SQL optimizations

We optimized the Targetprocess query generator to automatically throw away redundant SQL JOINs when possible. Currently, about 30-40% of queries are optimized with this approach, with a total of 10% of JOINs removed across all queries.

Business rules optimizations

We removed some redundant validations when creating or updating Entities in Targetprocess, and optimized some inefficient ones, which made API v1, Metrics, and Automation Rules several times faster in specific scenarios.

Context API optimizations

We made significant improvements to Context API (also known as Team-Project selector), it's now up to 30 times faster for large customers.

What’s Next

As well as having lots already in progress, there are also plenty of things we want to work on but haven’t got to yet: enabling HTTP/2, optimizing domain schema, improving custom fields performance, and optimizing comet configuration for better throughput. For our customers, this means Targetprocess is just going to keep getting better and better.