Sergey # Blog

Few thoughts about the current.

LINQ to Application Insights Traces

Application Insights has 7 days retention policy. You can run quite complicated queries using Metric and Search explorers but if you need to analyze historical data you need to use Continuous Export feature. You can read more about setting up Continuous Exports here.

Now what data is stored in Azure Blob Storage you can would want to query it. Tx (LINQ to Logs and Traces) library allows you to run Language Integrated Query (LINQ) directly on raw event sources: ad-hoc query on past history in trace and log files and standing queries on real-time feeds, such as Event Tracing for Windows (ETW) sessions. In the LINQPad experience of Tx is as if all the events were in a database, except, no database is involved - the query happens directly on raw logs/traces. And now Tx supports Application Insights data format.

There are 2 Nuget packages to work with Application Insights data:

Benefits of this approach are:

  • You can query historical data using well-known LINQ language
  • You can query Application Insights data with data from different streams (Tx also supports IIS logs, ETW traces and other trace formats.)
  • Tx is optimized for evaluating single-pass queries for multiplexed streams.
  • For more see Features of Playback.

Examples

Simple way to query from a local folder

How to setup ApplicationInsights

  • Add Tx.ApplicationInsights nuget package to your project. See this for details.
  • Add the following namespaces to a class: System.Reactive and Tx.ApplicationInsights.TelemetryType
  • Copy and paste the following code:
1
2
3
4
5
ApplicationInsightsFileStorage 
  .ReadFromFiles<CustomEvent>( 
      @"C:\Data\AppInsights\app1_6017e369258e42cc8cee20cd1956c3d2", 
      @"C:\Data\AppInsights\app2_4e49badc530d4187acc4c42e5e7d9ebe")    
  .SelectMany(i => i.@event);
  • Specify correct folder names. You can any number of folders or one parent folder
  • Hit F5 and get your results

More complicated way to query from a local folder

As explained earlier ApplicationInsightsFileStorage.ReadFromFiles is just a convenient method that you can use if you do not need to create complicated correlation queries. Same results you would get with this code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
var playback = new Playback();

playback.AddApplicationInsightsFiles(
  @"" // <- put here collection of directory paths containing .blob files
);

var query = playback.GetObservable<CustomEvent>().SelectMany(i => i.@event);

// If you are using LINQPad you could just dump the results
// query.Dump();

// Otherwise you could print them to console.
query
  .Subscribe(i => Console.WriteLine(i));

playback.Run();

Cache files from Azure blob storage

  • Add Tx.ApplicationInsights.Azure nuget package to your project. See this for details.
  • Add the following namespaces to a class: System.Reactive
  • Copy and paste the following code:
1
2
3
4
5
ApplicationInsightsAzureStorage
  .Cache(
      @"DefaultEndpointsProtocol=https;AccountName=#accountName#;AccountKey=#accountKey#",
      @"#containerName#",
      @"C:\AppInsights\");
  • Run the query (F5), then ApplicationInsightsStorage will fetch blob storage container and will iterate through stored there files and cache them into specified local folder if needed.

Simple way to query from Azure blob storage

There is a way how you could download and immediately query files from configurd Azure storage.

  • Add Tx.ApplicationInsights.Azure nuget package to your project. See this for details.
  • Add the following namespaces to a class: System.Reactive, Tx.ApplicationInsights and Tx.ApplicationInsights.TelemetryType
  • Copy and paste the following code:
1
2
3
4
5
6
ApplicationInsightsAzureStorage
  .ReadFromFiles<CustomEvent>(
      @"DefaultEndpointsProtocol=https;AccountName=#accountName#;AccountKey=#accountKey#",
      @"#containerName#",
      @"C:\AppInsights\")
  .Where(item => item.Context.Operation.Id == "12345");
  • Hit F5, then ApplicationInsightsStorage will fetch blob storage container and will iterate through stored there files and cache them into specified local folder if needed, then it reads and parses those files to return custom events with specified operation id.

More complicated way to query from Azure blob storage

Under the cover ApplicationInsightsAzureStorage is doing some work for you. You will get the same results if you write:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
var playback = new Playback();

playback.AddApplicationInsightsStorage(
  @"DefaultEndpointsProtocol=https;AccountName=#accountName#;AccountKey=#accountKey#", 
  @"#containerName#",
  @"C:\AppInsights\");

var query = playback
  .GetObservable<CustomEvent>()
  .Where(item => item.Context.Operation.Id == "12345");

// If you are using LINQPad you could just dump the results
// query.Dump();

// Otherwise you could print them to console.
query
  .Subscribe(i => Console.WriteLine(i));

playback.Run();
  • Results will be the same as in the first example but this approach is more flexible. We will look into more complicated example that uses buffers for which you need Observables.

Correlate Performance counters and Request

And here is the most advanced sample. It demonstrates how to write queries which use several Application Insights telemetry types, buffers and schedulers. We assume that you understand Rx, Observables and Tx to understand this sample. This code will return all requests that happened when performance counter “CPU % Total” value is more that 70 during 5 minutes interval.

  • Add Tx.ApplicationInsights nuget package to your project. See this for details.
  • Add the following namespaces to a class: System.Reactive, System.Reactive.Linq and Tx.ApplicationInsights.TelemetryType
  • Copy and paste the following code:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
var playback = new Playback();

playback.AddApplicationInsightsFiles(
  @"" // <- put here collection of directory paths containing .blob files
);

var query = playback
  .GetObservable<RequestEvent>()
  .Select(requestEvent => (BaseEvent)requestEvent)
  .Merge(playback
          .GetObservable<PerformanceCountersEvent>()
          .Select(performanceCountersEvent => (BaseEvent)performanceCountersEvent), 
      playback.Scheduler)
  .Buffer(TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(5), playback.Scheduler)
  .Where(buffer => buffer.Length > 0 && buffer.OfType<RequestEvent>().Count() > 0 && buffer.OfType<PerformanceCountersEvent>().Count() > 0)
  .Select(buffer =>
  {
      var cpuTotals = buffer
          .OfType<PerformanceCountersEvent>()
          .SelectMany(performanceCountersEvent => performanceCountersEvent.PerformanceCounters)
          .SelectMany(performanceCounter =>
          {
              return performanceCounter.Values
                  .Select(i => new 
                  {
                      Name = string.Format(
                          @"{0}/{1}/{2}"
                          performanceCounter.CategoryName,
                          performanceCounter.InstanceName,
                          i.Key),
                      Value = i.Value
                  })
              
          })
          .Where(metric => metric.Name == @"Processor/_Total/percentage_processor_total")
          .Select(metric => metric.Value)
          .ToArray();

      if (cpuTotals.Length > 0 && cpuTotals.Average() > 70)
      {
          return new
          {
              Requests = buffer
                  .OfType<RequestEvent>()
                  .ToArray(),
              CpuTotals = cpuTotals
          }
      }

      return null;
  })
  .Where(signal => signal != null);

// If you are using LINQPad you could just dump the results
// query.Dump();

// Otherwise you could print them to console.
query
  .Subscribe(i => Console.WriteLine(i));

playback.Run(); 

This sample is rather complex, let’s split it into several fragments:

As a first step we need to create Playback instance and register input source for it:

1
2
3
4
5
6
var playback = new Playback();

// Initialize input source for playback
playback.AddApplicationInsightsFiles(
  @"" // <- put here collection of directory paths containing .blob files
);

Then we need to created merged stream of request and performance counter telemetry event streams:

1
2
3
4
5
6
7
var mergedStream = playback
  .GetObservable<RequestEvent>()
  .Select(requestEvent => (BaseEvent)requestEvent)
  .Merge(playback
          .GetObservable<PerformanceCountersEvent>()
          .Select(performanceCountersEvent => (BaseEvent)performanceCountersEvent), 
      playback.Scheduler);

Next step is to apply hopping time window, which produces 5 minutes long batches of events:

1
2
var batches = mergedStream
  .Buffer(TimeSpan.FromMinutes(10), TimeSpan.FromMinutes(5), playback.Scheduler);

After that we could apply our custom logic for individual batches:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
var query = batches
  .Select(buffer =>
  {
      var cpuTotals = buffer
          .OfType<PerformanceCountersEvent>()
          .SelectMany(performanceCountersEvent => performanceCountersEvent.PerformanceCounters)
          .SelectMany(performanceCounter =>
          {
              return performanceCounter.Values
                  .Select(i => new 
                  {
                      Name = string.Format(
                          @"{0}/{1}/{2}"
                          performanceCounter.CategoryName,
                          performanceCounter.InstanceName,
                          i.Key),
                      Value = i.Value
                  })
              
          })
          .Where(metric => metric.Name == @"Processor/_Total/percentage_processor_total")
          .Select(metric => metric.Value)
          .ToArray();

      if (cpuTotals.Length > 0 && cpuTotals.Average() > 70)
      {
          return new
          {
              Requests = buffer
                  .OfType<RequestEvent>()
                  .ToArray(),
              CpuTotals = cpuTotals
          }
      }

      return null;
  })
  .Where(signal => signal != null);

And the last step is to use convinient Dump method on result query and start the playback.

1
2
3
query.Dump();

playback.Run(); 

You could find more samples here, in the official GitHub repository of this project.

Summary

Now you know how you could query Application Insights Continious Export telemetry files.

Comments