# Kinesis Agent for Windows Configuration Examples


 The `appsettings.json` configuration file is a JSON document that controls how Amazon Kinesis Agent for Microsoft Windows collects logs, events, and metrics. It also controls how Kinesis Agent for Windows transforms that data and streams it to various AWS services. For details about the source, sink, and pipe declarations in the configuration file, see [Source Declarations](source-object-declarations.md), [Sink Declarations](sink-object-declarations.md), and [Pipe Declarations](pipe-object-declarations.md). 

The following sections contain examples of configuration files for several different kinds of scenarios. 

**Topics**
+ [

## Streaming from Various Sources to Kinesis Data Streams
](#configuring-kaw-examples-sources)
+ [

## Streaming from the Windows Application Event Log to Sinks
](#configuring-kaw-examples-sinks)
+ [

## Using Pipes
](#configuring-kaw-examples-pipes)
+ [

## Using Multiple Sources and Pipes
](#configuring-kaw-examples-multiple)

## Streaming from Various Sources to Kinesis Data Streams


The following example `appsettings.json` configuration files demonstrate streaming logs and events from various sources to Kinesis Data Streams and from Windows performance counters to Amazon CloudWatch metrics.

### `DirectorySource`, `SysLog` Record Parser


The following file streams syslog format log records from all files with a `.log` file extension in the `C:\LogSource\` directory to the `SyslogKinesisDataStream` Kinesis Data Streams stream in the us-east-1 Region. A bookmark is established to ensure that all data from the log files is sent even if the agent is shut down and restarted later. A custom application can read and process the records from the `SyslogKinesisDataStream` stream.

```
{
  "Sources": [
    {
      "Id": "SyslogDirectorySource",
      "SourceType": "DirectorySource",
      "Directory": "C:\\LogSource\\",
      "FileNameFilter": "*.log",
      "RecordParser": "SysLog",
      "TimeZoneKind": "UTC",
      "InitialPosition": "Bookmark"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "SyslogKinesisDataStream",
      "Region": "us-east-1"
    }
  ],
  "Pipes": [
    {
      "Id": "SyslogDS2KSSink",
      "SourceRef": "SyslogDirectorySource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}
```

### `DirectorySource`, `SingleLineJson` Record Parser


The following file streams JSON-formatted log records from all files with a `.log` file extension in the `C:\LogSource\` directory to the `JsonKinesisDataStream` Kinesis Data Streams stream in the us-east-1 Region. Before streaming, key-value pairs for the `ComputerName` and `DT` keys are added to each JSON object, with values for the computer name and the date and time the record is processed. A custom application can read and process the records from the `JsonKinesisDataStream` stream. 

```
{
  "Sources": [
    {
      "Id": "JsonLogSource",
      "SourceType": "DirectorySource",
      "RecordParser": "SingleLineJson",
      "Directory": "C:\\LogSource\\",
      "FileNameFilter": "*.log",
      "InitialPosition": 0
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "JsonKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json",
      "ObjectDecoration": "ComputerName={ComputerName};DT={timestamp:yyyy-MM-dd HH:mm:ss}"
    }
  ],
  "Pipes": [
    {
      "Id": "JsonLogSourceToKinesisStreamSink",
      "SourceRef": "JsonLogSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}
```

### `ExchangeLogSource`


The following file streams log records generated by Microsoft Exchange and stored in files with the `.log` extension in the `C:\temp\ExchangeLog\` directory to the `ExchangeKinesisDataStream` Kinesis data stream in the us-east-1 Region in JSON format. Although the Exchange logs are not in JSON format, Kinesis Agent for Windows can parse the logs and transform them to JSON. Before streaming, key-value pairs for the `ComputerName` and `DT` keys are added to each JSON object containing values for the computer name and the date and time the record is processed. A custom application can read and process the records from the `ExchangeKinesisDataStream` stream. 

```
{
  "Sources": [
    {
       "Id": "ExchangeSource",
       "SourceType": "ExchangeLogSource",
       "Directory": "C:\\temp\\ExchangeLog\",
       "FileNameFilter": "*.log"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "ExchangeKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json",
      "ObjectDecoration": "ComputerName={ComputerName};DT={timestamp:yyyy-MM-dd HH:mm:ss}"
    }
  ],
  "Pipes": [
    {
      "Id": "ExchangeSourceToKinesisStreamSink",
      "SourceRef": "ExchangeSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}
```

### `W3SVCLogSource`


The following file streams Internet Information Services (IIS) for Windows log records stored in the standard location for those files to the `IISKinesisDataStream` Kinesis Data Streams stream in the us-east-1 Region. A custom application can read and process the records from the `IISKinesisDataStream` stream. IIS is a web server for Windows. 

```
{
  "Sources": [
    {
       "Id": "IISLogSource",
       "SourceType": "W3SVCLogSource",
       "Directory": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
       "FileNameFilter": "*.log"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "IISKinesisDataStream",
      "Region": "us-east-1"
    }
  ],
  "Pipes": [
    {
      "Id": "IISLogSourceToKinesisStreamSink",
      "SourceRef": "IISLogSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}
```

### `WindowsEventLogSource` with Query


The following file streams log events from the Windows system event log that have a level of `Critical` or `Error` (less than or equal to 2) to the `SystemKinesisDataStream` Kinesis data stream in the us-east-1 Region in JSON format. A custom application can read and process the records from the `SystemKinesisDataStream` stream. 

```
{
  "Sources": [
    {
         "Id": "SystemLogSource",
         "SourceType": "WindowsEventLogSource",
         "LogName": "System",
         "Query": "*[System/Level<=2]"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "SystemKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json"
    }
  ],
  "Pipes": [
    {
      "Id": "SLSourceToKSSink",
      "SourceRef": "SystemLogSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}
```

### `WindowsETWEventSource`


The following file streams Microsoft Common Language Runtime (CLR) exception and security events to the `ClrKinesisDataStream` Kinesis data stream in the us-east-1 Region in JSON format. A custom application can read and process the records from the `ClrKinesisDataStream` stream. 

```
{
  "Sources": [
    {
       "Id": "ClrETWEventSource",
       "SourceType": "WindowsETWEventSource",
       "ProviderName": "Microsoft-Windows-DotNETRuntime",
       "TraceLevel": "Verbose",
       "MatchAnyKeyword": "0x00008000, 0x00000400"
    }
  ],
  "Sinks": [
    {
      "Id": "KinesisStreamSink",
      "SinkType": "KinesisStream",
      "StreamName": "ClrKinesisDataStream",
      "Region": "us-east-1",
      "Format": "json"
    }
  ],
  "Pipes": [
    {
      "Id": "ETWSourceToKSSink",
      "SourceRef": "ClrETWEventSource",
      "SinkRef": "KinesisStreamSink"
    }
  ]
}
```

### `WindowsPerformanceCounterSource`


The following file streams performance counters for total files open, total login attempts since reboot, number of disk reads per second, and percentage of free disk space to CloudWatch metrics in the us-east-1 Region. You can graph these metrics in CloudWatch, build dashboards from the graphs, and set alarms that send notifications when thresholds are exceeded. 

```
{
  "Sources": [
    {
      "Id": "PerformanceCounter",
      "SourceType": "WindowsPerformanceCounterSource",
      "Categories": [
        {
          "Category": "Server",
          "Counters": [
            "Files Open",
            "Logon Total"
          ]
        },
        {
          "Category": "LogicalDisk",
          "Instances": "*",
          "Counters": [
            "% Free Space",
            {
              "Counter": "Disk Reads/sec",
              "Unit": "Count/Second"
            }
          ]
        }
      ],
    }
  ],
  "Sinks": [
    {
      "Namespace": "MyServiceMetrics",
      "Region": "us-east-1",
      "Id": "CloudWatchSink",
      "SinkType": "CloudWatch"
    }
  ],
  "Pipes": [
    {
      "Id": "PerformanceCounterToCloudWatch",
      "SourceRef": "PerformanceCounter",
      "SinkRef": "CloudWatchSink"
    }
  ]
}
```

## Streaming from the Windows Application Event Log to Sinks


The following example `appsettings.json` configuration files demonstrate streaming Windows application event logs to various sinks in Amazon Kinesis Agent for Microsoft Windows. For examples of using the `KinesisStream` and `CloudWatch` sink types, see [Streaming from Various Sources to Kinesis Data Streams](#configuring-kaw-examples-sources).

### `KinesisFirehose`


The following file streams `Critical` or `Error` Windows application log events to the `WindowsLogFirehoseDeliveryStream` Firehose delivery stream in the us-east-1 Region. If connectivity to Firehose is interrupted, events are first queued in memory. Then if necessary, they are queued to a file on disk until connectivity is restored. Then events are unqueued and sent followed by any new events.

You can configure Firehose to store the streamed data to several different kinds of storage and analysis services based on data pipeline requirements. 

```
{
  "Sources": [
    {
         "Id": "ApplicationLogSource",
         "SourceType": "WindowsEventLogSource",
         "LogName": "Application",
         "Query": "*[System/Level<=2]"
    }
  ],
  "Sinks": [
    {
       "Id": "WindowsLogKinesisFirehoseSink",
       "SinkType": "KinesisFirehose",
       "StreamName": "WindowsLogFirehoseDeliveryStream",
       "Region": "us-east-1",
       "QueueType": "file"
    }  
    ],
  "Pipes": [
    {
      "Id": "ALSource2ALKFSink",
      "SourceRef": "ApplicationLogSource",
      "SinkRef": "WindowsLogKinesisFirehoseSink"
    }
  ]
}
```

### `CloudWatchLogs`


The following file streams `Critical` or `Error` Windows application log events to CloudWatch Logs log streams in the `MyServiceApplicationLog-Group` log group. The name of each stream begins with `Stream-`. It ends with the four-digit year, two-digit month, and two-digit day that the stream was created, all concatenated (for example, `Stream-20180501` is the stream created on May 1, 2018). 

```
{
  "Sources": [
    {
         "Id": "ApplicationLogSource",
         "SourceType": "WindowsEventLogSource",
         "LogName": "Application",
         "Query": "*[System/Level<=2]"
    }
  ],
  "Sinks": [
    {
      "Id": "CloudWatchLogsSink",
      "SinkType": "CloudWatchLogs",
      "LogGroup": "MyServiceApplicationLog-Group",
      "LogStream": "Stream-{timestamp:yyyyMMdd}",
      "Region": "us-east-1",
      "Format": "json"
    }
  ],
  "Pipes": [
    {
      "Id": "ALSource2CWLSink",
      "SourceRef": "ApplicationLogSource",
      "SinkRef": "CloudWatchLogsSink"
    }
  ]
}
```

## Using Pipes


The following example `appsettings.json` configuration file demonstrates using pipe-related features.

 This example streams log entries from the `c:\LogSource\` to the `ApplicationLogFirehoseDeliveryStream` Firehose delivery stream. It includes only lines that match the regular expression specified by the `FilterPattern` key-value pair. Specifically, only lines in the log file that start with `10` or `11` are streamed to Firehose. 

```
{
  "Sources": [
    {
      "Id": "ApplicationLogSource",
      "SourceType": "DirectorySource",
      "Directory": "C:\\LogSource\\",
      "FileNameFilter": "*.log",
      "RecordParser": "SingleLine"
    }
  ],
  "Sinks": [
    {
       "Id": "ApplicationLogKinesisFirehoseSink",
       "SinkType": "KinesisFirehose",
       "StreamName": "ApplicationLogFirehoseDeliveryStream",
       "Region": "us-east-1"
    }  
    ],
  "Pipes": [
    {
      "Id": "ALSourceToALKFSink",
      "Type": "RegexFilterPipe",
      "SourceRef": "ApplicationLogSource",
      "SinkRef": "ApplicationLogKinesisFirehoseSink",
      "FilterPattern": "^(10|11),.*"
    }
  ]
}
```

## Using Multiple Sources and Pipes


The following example `appsettings.json` configuration file demonstrates using multiple sources and pipes.

This example streams the application, security, and system Windows Event Logs to the `EventLogStream` Firehose delivery stream using three sources, three pipes, and a single sink.

```
{
    "Sources": [
		{
		  "Id": "ApplicationLog",
		  "SourceType": "WindowsEventLogSource",
		  "LogName": "Application"
		},
		{
		  "Id": "SecurityLog",
		  "SourceType": "WindowsEventLogSource",
		  "LogName": "Security"
		},
		{
		  "Id": "SystemLog",
		  "SourceType": "WindowsEventLogSource",
		  "LogName": "System"
		}
    ],
    "Sinks": [
		{
		  "Id": "EventLogSink",
		  "SinkType": "KinesisFirehose",
		  "StreamName": "EventLogStream",
		  "Format": "json"
		},
    ],
    "Pipes": [
		{
		  "Id": "ApplicationLogToFirehose",
		  "SourceRef": "ApplicationLog",
		  "SinkRef": "EventLogSink"
		},
		{
		  "Id": "SecurityLogToFirehose",
		  "SourceRef": "SecurityLog",
		  "SinkRef": "EventLogSink"
		},
		{
		  "Id": "SystemLogToFirehose",
		  "SourceRef": "SystemLog",
		  "SinkRef": "EventLogSink"
		}
    ]
}
```