Today, I’d like to talk about using Microsoft Sentinel and address a common question that many teams have when they are starting to work with the Sentinel SIEM/SOAR solution….Where is my Data?
A common challenge that teams new to Sentinel often face is determining if all the monitoring data from various sources is actually getting into your Log Analytics workspace. Where is the data stored? How do we determine if the data is landing in our storage account?
First, let’s get logged into your Azure tenant at https://portal.azure.com and locate your Log Analytics Workspace.
In the top navigation bar search for Log Analytics Workspace. Choose the LAW result and locate your account used for Sentinel testing (We are being non-destructive today, but we should always practice in Non-Prod!).
Once in your Log Analytics Workspace, open the blade Logs under the General section. This will load up with a great Queries screen that can provide some excellent pre-populated queries for you as shortcuts to get querying your data quickly.
Click on Security > then scroll down a little bit until you see Threat Intelligence Logs.
Click on Load to Editor to use the Threat Intelligence Query all pre-loaded & ready to run against your data, or close the Queries screen when you’re ready.
Once your done with that fun query, let’s get back on track!
Another way to get there is to expand your Tables in Log Analytics Workspace, then find the table you want to work with and double-click it. This will populate that table name into your query window so you car run the query.
I would like to encourage you to explore your data tables a little bit, just in general really. It’s a great way to start learning all the table structures, data stored in each table, and can really help to build your KQL and knowledge of your own data. This all supports being a better Analyst, threat hunter, and then you can share that knowledge with your team!
Let’s jump into our Log Analytics Workspace and start exploring together!
Now we are ready to explore running a query.
Let’s focus on our original question — Where is my Data?
The data that comes into your Log Analytics workspace will be stored in one of the tables that you see listed under Logs.
Let’s look at an example of how to find what log agents are reporting:
- Open a new query tab
- Build your query – Under LogManagement, select Heartbeat (table is Heartbeat). You can copy this simple query which will show us distinct computers from the table Heartbeat.
- Set the time range for the query – 24 hours or less is a great place to start. If you have a lot of log collectors, or machine agents reporting, consider going lower to 4 hours to start.
- Run your query to see your results!
- …..well….your results are in! Check here to see if a heartbeat from that log collector or endpoint is reporting.
So, now we know what systems are reporting into our Log Analytics workspace.
Next, we should see if we can narrow down if all the event logs (or other logs such as firewall logs, etc) are reporting into our storage account. Let’s explore a bit more…..
Ok, so this time, less arrows and let’s see how we all do exploring our data…..
This time, I’ve added the SecurityEvent table, then filtered my results with just one Computer (fieldname) by it’s name from our first query on the HeartBeat table. This ensures that the computer is reporting a heartbeat back to Azure and is connecting, now we can see data fed from local events into SecurityEvents table in our Log Analytics Workspace.
We can see in the screenshot above that the the computer is successfully reporting local events from the event log into our Azure storage account. This means that we can now use the magical powers of Microsoft Sentinel Analytics Rules to build some logic and alerting around this data. That part will be another day, but for today, let’s make sure we’ve solved for all our questions:
Where is my data?
– We found it in our Log Analytics Workspace.
How do we determine if all our data is landing in Azure?
-We can take the SecurityEvents data filtered by one Computer name and compare to local results to ensure all our intended events are getting in to where we need them, in Azure Log Analytics Workspace.
In many cases, remember that our collector or agent endpoint configurations may filter events ‘locally’ before they are moved up to Azure. You may not see every single event as a “like for like” unless that is what you have configured. Filtering these events is always a great idea, I can only say to follow your organizational guidance in this case to ensure that you capture the correct events. We could spend a whole series of write-ups discussing just that part!
Thanks for joining in with me to take a closer look at our logging data, have a bit of fun with KQL, and working through a little bit of validation for our Log Analytics data.
Remember to keep exploring that data, and really try to gain an understanding of what is actually coming into your Azure storage so you can make sure that you are capturing what you need, but also keeping the useful pieces of data collected.
