Trasier's mission is to bring Distributed Tracing to the next level and help you to gain unknown insights into your processes by enabling
Business Tracing.

Introduction

Trasier is a Busines Tracing as a service system build on top of Open Tracing standard.

Distributed Tracing

It is a method used to monitor and profile applications in a distributed environment. It gives insights into the microservices architecture and helps to track down technical issues.

Benefits of distributed tracing

  • Latency visualization

  • Reveal the real architecture (shows dependencies between services)

  • Error analysis

  • Infrastructure check

High-level architecture

The diagram below depicts the components needed to implement distributed tracing concepts.

500

Trace-Context (Specification)

Services need to understand the data they exchange with each other. To achieve this we need to have some kind of specification or API. In distributed tracing, the specification is the Trace-Context.

Trace-Context consists of Trace-ID and Span-ID which are propagated between services with every request.

Context Propagation

There must be a mechanism to transfer the Trace-Context between services and within one service itself from the endpoint to the client-side.

There are two ways to achieve this:

  • Explicit - as part of the service or method API

  • Implicit - using service instrumentation

There are several instrumentation libraries like Open Tracing or Open Census to achieve this. The context information within the service is usually propagated with help of a ThreadLocal, while context between services is usually propagated within the headers (for example HTTP headers).

Decoupled collector - Tracer

The collector (Tracer or a Tracer Client) is usually an agent or instrumentation library that collects the messages and sends them asynchronously to the tracing backend.

Collector and Datastore

On the backend side, there must be a service or a collector that will receive, process, and store the data. This might be for example a queuing system or a simple service that sends the data to the datastore.

User interface

The data has to be presented to the user in some kind of way.

There are a few projects compatible with Open Tracing like Jaeger and Zipkin and a large amount of APM providers can nowadays visualize distributed tracing data.

Distributed tracing vocabulary

500

Span

Span is an operation that was executed. Typically it is the communication of two services. Span carries all the necessary information needed for further analysis. This is for example operation the name (mandatory), start/end timestamps, error codes (in the response), headers, log entries, etc.

Spans can have parent spans. In the picture above we can see three spans, two root spans (requestOffer, bookOffer), and a child span (checkPayment as a child of bookOffer span).

A Span has a unique Id called Span-Id which is part of the Trace-Context.

Trace

Trace is a request to the system that goes through all services. It has a collection of spans. Trace has a unique Id called the Trace-Id and is part of the Trace-Context.

In the example above we can see two traces, once initialized by the offerRequest and one initialized by the bookingRequest.

Trace Context

Consist of the Trace-Id and Span-Id. The trace context must be propagated with every call between services as well as within a service itself.

Business Tracing

Business Tracing takes Distributed Tracing to the next level by focusing on the business aspects of your applications. It correlates and processes the communication of your business applications in a way that helps to track down business issues.

Conversation

Instead of tracking technical requests, we are tracing processes from a business perspective.

To track the business process we need to add the Conversation-Id to the Trace-Context.

A Conversation is a collection of Traces. Typically the request initiator (Website or other UI) knows when a business process starts and when it ends and must generate proper `Conversation-Id`s.

In the example below the business process is a booking process. It starts with an offer request, goes through payment, ticketing, email sending, etc. The UI knows that for example a new login starts a new business process and email sending may end a business process.

500

With the help of Converstaion, we can trace the whole business process together with message payloads from the very beginning to the very end. Also, asynchronous batch processing can be visible along with a business process if the Conversation-Id is known to the batch processor.

500

Message payloads

Unlike Distributed Tracing tools, Trasier can store message payloads (incoming and outgoing) within the span. Storing message payloads means that Trasier does not sample requests like other distributed tracing tools do.

Note that not all messages must be intercepted and stored. Trasier offers the ability to configure the system to not collect sensitive data (like customer data or payment information).

Use cases

Business Tracing

There are lots of use cases for Business Tracing.

Imagine a flight reservation system and a customer complaining that he booked a ticket to Sydney (Australia) but got a ticket to Sydney (Canada).

In flight reservation systems, the trip is selected at the very beginning of the business process and the ticket is produced at the very end. Was this a user error (the user didn’t notice that he picked the wrong destination) or was it a system error - incorrect data printed on the ticket?

Imagine another customer claims that he was offered a flight for $200 but was charged $300. How to analyze this issue in a system where prices are dynamically changing? With Trasier - Business Tracing we would be able to find the conversation based on the customer’s booking reference and check what offers were send to his browser in the past.

500

Bug triage

Being able to quickly see where an error (technical or business error) comes from makes it easier to assign a bug to the appropriate team.

Mocking and replaying

Having all the message payloads stored opens new possibilities for the development teams.

  • Mocking - instead of using fake objects or storing XML files which are hard to maintain, one could use a conversation id of a business proces to read all the responses from the tracing backend. If the system gets updated (new API), it is enough to replace the conversation id.

500

  • Replaying - instead of explaining the steps to reproduce a bug and manually re-enter all the details in some kind of UI (or creating tests to reproduce the error) one could with the help of conversation id read the requests send to the system, and re-send them once again to the locally deployed instance and immediately start debugging.

500

Predictive analysis

The data stored by the Trasier system is the perfect foundation for extensive research and data analysis. Gathering the equivalent information from your existing data stores is expensive and may not even be possible. The chance is to finally make take the step to predictive analysis on virtual processes.

Possible use cases:

  • Anomaly detection

    • Higher number of errors

    • Slower response times

    • Revenue drop

    • Ticket prices suddenly offered at a very low price

  • Business values

    • How much users typically are willing to pay for service upgrades

  • On the fly verifications

    • Validate on the fly that the amount paid was send to the user as email confirmation (email service), to the SAP Service, to the payment provider.

How does Trasier work

Trasier uses an instrumentation library as a Tracing Client, which intercepts the communication between services and takes care of the context propagation. For that purpose, Trasier is injecting and extracting HTTP Headers.

HTTP Headers

Table 1. Trasier client supported headers

X-Conversation-Id

Identifies the business process. This id remains the same until the business process ends.

X-Trace-Id

Identifies the trace. The same trace id will be used from the moment when the request was initiated through all the calls to the different backends. Trasier client creates a new Trace-Id if the header is missing in the request.

X-Span-Id

Identifies the span, for example, a single call between Service A and Service B. Trasier client creates a new Span-Id if the header is missing in the request.

X-Incoming-Endpoint-Name

Used to identify the calling system. If the header is missing, a 'UNKNOWN_IN' will be displayed in the Trasier UI.

X-Conversation-Sample

Indicates whether to send the data to the tracing backend. Default is true. If the sample flag is set to false, no underlying spans will be intercepted.

Data processing

The intercepted messages are sent to the Trasier backend which uses a high-performance processing pipeline to process, validate, index, and store the data. Note that the data are stored encrypted, but the indexes are not. That’s why the user has to configure Trasier (using provided configuration options or by writing an interceptor) not to send sensitive data to the backend.

The data send to Trasier is indexed, meaning the user can search for phrases that occur in message payloads, operation names, or message headers. Note that the data is optimized for indexing, so that XML tags, Base64 encodings, images, etc are stripped.

Registration

To register go to https://trasier.com/#/register. After filling in the registration form an activation email will be sent to the given email address. During the activation, the user will be asked to set up a new password.

After the account was activated, an email with the account id will be sent.

From now on it is possible to access the Trasier UI at https://ui.trasier.com.

Note
Either the email address or the account id can be used for logging in.

After logging in, user will be asked to configure spaces.

Configuring spaces

Space defines a storage for the data that is sent to the tracing backend.

A space like an environment, for example dev, test or production are reasonable space names.

If an organization has multiple distinct applications that are traced independently, the space can be prefixed: bookingapp_dev iottrace_dev.

Note that it is not possible to link data between spaces, i.e. data stored in space prod is not visible in space test.

Once the space was created, a configuration details such as account id, client id, and client secret can be displayed. This is needed for authenticating the tracing client.

spaces

Note
It is possible to delete a space. This operation cannot be undone and all previously stored data in this space will be removed.

Trasier Integration

Authentication

Trasier uses the standard Oauth2 authentication with client id and client secret and every space has its own configuration.

The Spring-based Trasier Java Client takes care of the authentication automatically. Other projects can use the OAuthTokenSafe as a reference implementation.

Java Projects

There is a dedicated client for Java. The Java client is a set of open-source libraries hosted on GitHub.

Configuration

Trasier client is build on top of OpenTracing. Both OpenTracing and Trasier must be enabled by a feature toggle:

opentracing.spring.web:
  enabled: ${TRASIER_ACTIVATED:true}
  client.enabled: ${TRASIER_ACTIVATED:true}
trasier:
  proxy:
    host: ${PROXY_HOST}
    port: ${PROXY_PORT}
  client:
    accountId: ${ACCOUNT_ID:1234}
    systemName: ${APP_NAME}
    activated: ${TRASIER_ACTIVATED:true}
    payloadTracingDisabled: ${TRASIER_PAYLOAD_TRACING_DISABLED:false}

Aditionally one must configure spaces to tell Trasier where to send the data:

trasier:
  client:
    spaceKey: ${SPACE_KEY}
    clientId: ${CLIENT_ID}
    clientSecret: ${CLIENT_SECRET}

Integration with spring

Either GRPC or REST protocol can be used to send the data into the tracing backend.

For GRPC add the following dependency to your pom.xml file:

pom.xml
<!-- Trasier Dependencies -->
<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-spring-web</artifactId>
    <version>2.1.3</version>
</dependency>
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-grpc</artifactId>
    <version>2.3.6</version>
</dependency>
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-interceptor</artifactId>
    <version>2.3.6</version>
</dependency>

For REST add the following dependency to your pom.xml file:

pom.xml
<!-- Trasier Dependencies -->
<dependency>
    <groupId>io.opentracing.contrib</groupId>
    <artifactId>opentracing-spring-web</artifactId>
    <version>2.1.3</version>
</dependency>
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-rest</artifactId>
    <version>2.3.6</version>
</dependency>
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-interceptor</artifactId>
    <version>2.3.6</version>
</dependency>

Add the following configs in the web.xml:

web.xml
<filter>
    <filter-name>TrasierBufferFilter</filter-name>
    <filter-class>com.trasier.opentracing.spring.interceptor.servlet.TrasierBufferFilter</filter-class>
</filter>
<filter-mapping>
    <filter-name>TrasierBufferFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>
<filter>
    <filter-name>TracingFilter</filter-name>
    <filter-class>io.opentracing.contrib.web.servlet.filter.TracingFilter</filter-class>
</filter>
<filter-mapping>
    <filter-name>TracingFilter</filter-name>
    <url-pattern>/*</url-pattern>
</filter-mapping>
<listener>
    <listener-class>com.trasier.opentracing.spring.interceptor.servlet.TrasierSpringConfigListener</listener-class>
</listener>

Add the following bean definitions:

<bean id="trasierClientConfiguration" class="com.trasier.client.configuration.TrasierClientConfiguration">
        <property name="accountId" value="${trasier.accountId}" />
        <property name="spaceKey" value="${trasier.spaceKey}" />
        <property name="clientId" value="${trasier.clientId}" />
        <property name="clientSecret" value="${trasier.clientSecret}" />
        <property name="systemName" value="${trasier.clientSecret}" />
        <property name="activated" value="${trasier.activated}" />
        <property name="payloadTracingDisabled" value="${trasier.payloadTracingDisabled}" />
</bean>

<bean id="trasierProxyConfiguration" class="com.trasier.client.configuration.TrasierProxyConfiguration">
        <property name="host" value="${trasier.proxyHost}" />
        <property name="port" value="${trasier.proxyPort}" />
</bean>

<bean id="trasierEndpointConfiguration" class="com.trasier.client.configuration.TrasierEndpointConfiguration" />

<bean id="trasierSampleByOperationConfiguration" class="com.trasier.client.spring.spancontrol.TrasierSampleByOperationConfiguration" />

<bean id="trasierScopeManager" class="com.trasier.client.opentracing.TrasierScopeManager">
</bean>

<bean id="trasierTracer" class="com.trasier.client.opentracing.TrasierTracer">
        <constructor-arg index="0" ref="trasierSpringClient" />
        <constructor-arg index="1" ref="trasierClientConfiguration" />
        <constructor-arg index="2" ref="trasierScopeManager" />
</bean>

<context:component-scan base-package="com.trasier.client.spring"/>
<context:component-scan base-package="com.trasier.opentracing.spring.interceptor"/>

Integration with spring boot

To use the GRPC protool to send the data into tracing backend add the following dependencies to your pom.xml file:

pom.xml
<!-- Trasier -->
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-grpc-starter</artifactId>
    <version>2.3.6</version>
</dependency>

To use the REST protool to send the data into tracing backend add the following dependencies to your pom.xml file:

pom.xml
<!-- Trasier -->
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-rest-starter</artifactId>
    <version>2.3.6</version>
</dependency>

A reference implementation can be found at https://github.com/trasiercom/springboot-example/tree/tracing/trasier.

Tracing CXF Soap Messages

Incomming SOAP calls are automatically traced by including the trasier-client-spring-rest-starter dependency. Tracing the Outgoing SOAP calls depends on the soap client implementation. Spring’s WebServiceTemplate is supported out of the box. For CXF the folowing dependency must be included:

pom.xml
<!-- Trasier -->
<dependency>
<dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-rest-starter</artifactId>
    <version>2.3.6</version>
</dependency>
    <groupId>com.trasier</groupId>
    <artifactId>trasier-client-spring-interceptor-ws</artifactId>
    <version>2.3.6</version>
</dependency>

In this case, the TracingSoapHandler must manually be added to the `BindingProvider`s handler chain. Example implementation:

BindingProvider bindingProvider = (BindingProvider) service;
Binding binding = ((BindingProvider) service).getBinding();
List<Handler> handlerChain = binding.getHandlerChain();
handlerChain.add(new TracingSoapHandler(trasierTracer, trasierClientConfiguration));
binding.setHandlerChain(handlerChain);

Span filter configuration

(since 2.3.6)

The Trasier client allows defining URL and operation name patterns to disable message payloads or skip entire spans from being traced. For this purpose, the TrasierFilterConfigurations bean is used, which is automatically created in Spring Boot applications.

Three modes can be used:

  • allow - only matching spans will be traced (works like a whitelist)

  • cancel - matching spans will be skipped (works like a blacklist)

  • disablePayload - matching spans will not contain message payload

Examples:

Trace only offers service requests

application.yaml (spring boot)
trasier:
  client:
    filters:
      allow:
        url: ".*offers.*"
system properties (alternative)
-Dtrasier.client.filters.allow.url=.*offers.*
xml configuration
<bean id="filterConfigurations" class="com.trasier.client.configuration.TrasierFilterConfigurations">
    <property name="allow" ref="allowFilter" />
</bean>

<bean id="allowFilter" class="com.trasier.client.configuration.TrasierFilterConfiguration">
    <property name="url" value="${trasier.client.filters.allow.url}" />
</bean>

Do not trace requests to the login and logout endpoint and disable payload for all payment service calls

application.yaml (spring boot)
trasier:
  client:
    filters:
      cancel:
        url: ".*login.*|.*logout.*"
      disablePayload:
        url: ".*payment.*"
system properties (alternative)
-Dtrasier.client.filters.cancel.url=.*login.*|.*logout.*
-Dtrasier.client.filters.disablePayload.url=.*payment.*
xml configuration
<bean id="filterConfigurations" class="com.trasier.client.configuration.TrasierFilterConfigurations">
    <property name="cancel" ref="cancelFilter" />
    <property name="disablePayload" ref="disablePayloadFilter" />
</bean>

<bean id="cancelFilter" class="com.trasier.client.configuration.TrasierFilterConfiguration">
    <property name="url" value="${trasier.client.filters.cancel.url}" />
</bean>

<bean id="disablePayloadFilter" class="com.trasier.client.configuration.TrasierFilterConfiguration">
    <property name="url" value="${trasier.client.filters.disablePayload.url}" />
</bean>

Do not trace DELETE methods

application.yaml (spring boot)
trasier:
  client:
    filters:
      cancel:
        operation: "DELETE"
system properties (alternative)
-Dtrasier.client.filters.cancel.operation=DELETE
xml configuration
<bean id="filterConfigurations" class="com.trasier.client.configuration.TrasierFilterConfigurations">
    <property name="cancel" ref="cancelFilter" />
</bean>

<bean id="cancelFilter" class="com.trasier.client.configuration.TrasierFilterConfiguration">
    <property name="url" value="${trasier.client.filters.cancel.operation}" />
</bean>

Tracing messages manually

One can manually write any message at any point in the application to the Trasier backend. This is useful if someone has an interceptor that is not yet supported by the library. There are also cases when it is important to see an internal state of the application along with the messages exchanged by services (complicated algorithms, filter logic, etc).

The following snippet demonstrates how to do that:

Filter.java
@Autowired
private Tracer tracer;

private void traceableFilterMethod(Filter filter, List<String> data) {

    io.opentracing.Span span = tracer.buildSpan("MY_OPERATION_NAME")
                .withTag(Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_CLIENT)
                .start();

    Span trasierSpan = ((TrasierSpan) span).unwrap();
    trasierSpan.setIncomingContentType(ContentType.TEXT);
    trasierSpan.setBeginProcessingTimestamp(System.currentTimeMillis());
    trasierSpan.setIncomingData(data.toString());

    Scope scope = this.tracer.activateSpan(span);

    filter.apply(data); // try-catch omitted for readability

    trasierSpan.setOutgoingContentType(ContentType.TEXT);
    trasierSpan.setFinishProcessingTimestamp(System.currentTimeMillis());
    trasierSpan.setOutgoingData(data.toString());
    trasierSpan.setStatus(TrasierConstants.STATUS_OK);

    scope.close();
    span.finish();

}

In a similar way we can implement a custom HTTP filter:

HttpFilter
@Autowired
private Tracer tracer;

@Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) {

    CachedServletRequestWrapper request = createCachedRequest((HttpServletRequest) servletRequest);
    CachedServletResponseWrapper response = createCachedResponse((HttpServletResponse) servletResponse);

    io.opentracing.Span span = tracer.buildSpan("MY_OPERATION_NAME")
                .withTag(Tags.SPAN_KIND.getKey(), Tags.SPAN_KIND_CLIENT)
                .start();

    // instance checks skipped for readability
    Span trasierSpan = trasierSpan = ((TrasierSpan) span).unwrap();
    currentSpan.setIncomingData(new String(request.getContentAsByteArray()));
    currentSpan.setIncomingHeader(getRequestHeaders(request));
    currentSpan.setIncomingContentType(ContentType.XML);
    currentSpan.setBeginProcessingTimestamp(System.currentTimeMillis());

    Scope scope = this.tracer.activateSpan(span);

    filterChain.doFilter(request, response); // try-catch omitted for readability

    trasierSpan.setOutgoingData(response.getContentAsByteArray());
    trasierSpan.setOutgoingHeader(extractHeaders(messageContext.getResponse()));
    trasierSpan.setOutgoingContentType(ContentType.XML);
    trasierSpan.setFinishProcessingTimestamp(System.currentTimeMillis());
    trasierSpan.setStatus(TrasierConstants.STATUS_OK);

    scope.close();
    span.finish();
}

Integration without spring

At the moment the Java Client is spring-based only. Further implementations are planned. By then one hast to handle the context propagation in the application on its own. The API, interceptors, and client interface can be taken from trasier-client-core and trasier-client-api. The OAuth can be implemented similarly as it is in OAuthTokenSafe.

Non-Java projects

At the moment there are no clients for languages other than Java. One must take care of the context propagation, auth and, sending data to the Trasier Backend.