While the title of this blog post says it is about how to add data to Microsoft Sentinel, technically the APIs we will discuss will write to Azure Monitor. However, the data will be readable by Microsoft Sentinel, so it is close enough 🙂
Microsoft Sentinel, through its extensive list of built-in data connectors, has a lot of ways to ingest data. But what happens if the system you are using doesn’t have a data connector? How do you get the data you need into Microsoft Sentinel so you can perform all the queries you need?
You could use the new Microsoft Sentinel Codeless Connector that is still in preview when this post was written. This appears to use the Azure Monitor HTTP Data connector (more information here) which is a quick and easy solution if the data is coming through properly formatted and you want everything that the API provides. Of course, the API may take parameters to filter out some of the data. More information on creating a codeless connector can be found here)
The second way is to use the new Logs ingestion API which is also in preview at this time. While there is a bit more configuration that needs to be done, including creating a new Azure AD Application, a data collection endpoint (DCE), and a data collection rule (DCR), you can do much more with it. Not only can you ingest the data and manipulate it, but you can also write to a small set of Microsoft Sentinel tables, as well as Basic tables. As the documentation provided here is very well done, I will not be recreating the code. Just follow the instructions in the documentation and you will be good to go.
What I will to is to attempt to show you the pros and cons of each way to ingest data to help you make your decision on which one to use.
BTW, both ways will allow you to specify a column to use to populate the TimeGenerated column.
HTTP Data Collector API
- Easy to setup. Just needs the Log Analytics workspace’s Workspace ID and key.
- Can only write to custom logs
- Columns created have “_<data type>” appended to them. Some people may not consider this an issue, but it bugs me.
- Data must be in the correct format when ingesting it. No changes to the data will be made by Azure Monitor.
- Cannot write to a basic table.
Logs ingestion API
- Can write to custom logs as well as the CommonSecurityLog, SecurityEvents, Syslog, and WindowsEvents tables
- Can add columns to existing tables (see above). Each custom column must end in “_CF”
- Columns created in a custom table do not have a suffix automatically added.
- Can use a DCR to modify data. This can be adding new columns, not including certain rows, and more.
- Can write to a Basic table. The table must be converted to Basic after it has been created.
- Requires more configuration. You need to setup an Azure AD application, a DCE, and a DCR. A DCE and DCR will be required for each feed although you can reuse the Azure AD Application.
As you can see, there are two different ways to ingest data into Microsoft Sentinel. For most of your needs, the HTTP Data Collector API will work just fine. However, if you need more control over your ingestion, consider using the Logs ingestion API.
Either way, you can easily add your own data into Microsoft Sentinel for your use.