Splunk

Introduction

KringleCon - Great Room

9. Splunk!

Difficulty: 3/5

Help Angel Candysalt solve the Splunk challenge in Santa's great hall. Fitzy Shortstack is in Santa's lobby, and he knows a few things about Splunk. What does Santa call you when you complete the analysis?

https://hhc21.bossworkshops.io/en-US/app/SA-hhc/santadocs

Conversation

Helpful Searches

Sysmon for Linux All events:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational

Process Creation:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=1

Network Connection:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=3

Using Splunk stats and sort commands to find most/least common value of a field:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=1 user=eddie
| stats count by CommandLine
| sort - count

GitHub Audit Log Events

index=main sourcetype=ghe_audit_log_monitoring

Webhook Events (Includes detailed vulnerability alerts)

index=main sourcetype=github_json

Tasks

Task 1

Capture the commands Eddie ran most often, starting with git. Looking only at his process launches as reported by Sysmon, record the most common git-related CommandLine that Eddie seemed to use.

Solution

First search I ran was for * so I could see what kind of data I had available to me. Sweet, looks like there are a few potentially interesting sources, including Journald:Microsoft-Windows-Sysom/Operational, something related to github enterprise, and githubwebhook.

Next I filtered just the source to the Sysmon one. I immediately saw that one of the Selected Fields was CommandLine.

I need to filter by user Eddie though, since the first result is user root. Checking out the user field confirmed the target username was 'eddie'.

I added the user to the search, then checked out the CommandLine field again, and saw that the most common git-related command that user eddie has run is 'git status'.

Correct! Query:

source="Journald:Microsoft-Windows-Sysmon/Operational" user=eddie

Task 2

Looking through the git commands Eddie ran, determine the remote repository that he configured as the origin for the 'partnerapi' repo. The correct one!

Solution

Ok, I searched Sysmon for commands from eddie, that related to git and origin. I saw that eddie tried adding the origin from remote https and remote ssh.

source="Journald:Microsoft-Windows-Sysmon/Operational" user=eddie "git" "origin"

I ran the search for the specific git remote add origin string and saw the two instances of the command that were run.

User eddie configured the https repo to the remote origin  first, at 9:41:05 PM, then configured the ssh repo to the remote origin at 9:42:38 PM, so the ssh remote repo must be the correct one.

source="Journald:Microsoft-Windows-Sysmon/Operational" user=eddie CommandLine="git remote add origin*"

Task 3

The 'partnerapi' project that Eddie worked on uses Docker. Gather the full docker command line that Eddie used to start the 'partnerapi' project on his workstation.

Solution

Ok, so they would have had to preface the thing with 'docker' something, so let's search that:

source="Journald:Microsoft-Windows-Sysmon/Operational" user=eddie CommandLine="docker *"

Oh, there's only 2 CommandLines.

Well that's obvious they used:

docker compose up

Task 4

Eddie had been testing automated static application security testing (SAST) in GitHub. Vulnerability reports have been coming into Splunk in JSON format via GitHub webhooks. Search all the events in the main index in Splunk and use the sourcetype field to locate these reports. Determine the URL of the vulnerable GitHub repository that the elves cloned for testing and document it here. You will need to search outside of Splunk (try GitHub) for the original name of the repository.

Solution

I need to search for sourcetype github_json.

I started at the bottom of the list. There were only 27 results so not that bad to go through.

A few items up, I see a created action logged. This action created a git folder from the html_url listed.

I went to check out https://github.com/elfnp3/dvws-node/ and saw that it was forked from: snoopysecurity/dvws-node.

Task 5

Santa asked Eddie to add a JavaScript library from NPM to the 'partnerapi' project. Determine the name of the library and record it here for our workshop documentation.

Solution

Ok, so I want to find a commit that includes some .js files. No that wasn't it.

I want to search the CommandLine for stuff that includes npm. 24 results!

 index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational CommandLine="*npm*"

This one might be it! The user installed express-fileupload.

That wasn't it. Maybe it's this node-gyp one.

Still not it, time to reassess. Let's look at the github audit log:

index=main sourcetype=ghe_audit_log_monitoring

Ah ok, here we can filter by repo, so I'll look at partnerapi.

index=main sourcetype=ghe_audit_log_monitoring repo="elfnp3/partnerapi"

Shit, no. This isn't that verbose, not nearly enough for me to see specific files etc.

What if I go back to the github_json sourcetype? Is there a repo field there? Yes, repository.name:

index=main sourcetype=github_json "repository.name"=partnerapi

The most recent result was a commit that added controllers/routetojack.js: Hm, maybe this one? Correct! I wound up searching the following:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational "node" user=eddie

This was because I realized that node was used to install npm modules after a quick google search, so I thought that if I could look at all the commands that included the word node, I might see something that helped me. I wound up seeing the holiday-utils-js package referenced directly in an install command, and figured that since I had tried some others that didn't work, and since it referenced the holidays and stuff, then it was probably the right library.

Task 6

Another elf started gathering a baseline of the network activity that Eddie generated. Start with their search and capture the full process_name field of anything that looks suspicious.

Their Search:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=3 user=eddie NOT dest_ip IN (127.0.0.*) NOT dest_port IN (22,53,80,443)
| stats count by dest_ip dest_port

Solution

First I switched from smart mode to verbose mode so I could see the exact events, since there were only 3.

There are only 2 values for process_name, and obviously the nc.openbsd one is the suspicious shit.

Task 7

Uh oh. This documentation exercise just turned into an investigation. Starting with the process identified in the previous task, look for additional suspicious commands launched by the same parent process. One thing to know about these Sysmon events is that Network connection events don't indicate the parent process ID, but Process creation events do! Determine the number of files that were accessed by a related process and record it here.

Solution

Ok, so I need to get the process id for nc.openbsd.

I used the given Process Creation search query, then added the nc process name:

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=1 process_name="/usr/bin/nc.openbsd"

Ok, that is the only process that pops, so let's see if there's a process id field. Yup, it's 6788.

I removed the process_name filter and added the parent_process_id filter.

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=1 parent_process_id=6788

That's it! The other suspicious command was dumping ssh key files and some aws credential file:

The command accessed 6 files.

Task 8

Use Splunk and Sysmon Process creation data to identify the name of the Bash script that accessed sensitive files and (likely) transmitted them to a remote IP address.

Solution

Ok, I'll change the parent_process_id filter to PID, keeping the same value to filter on, since this should give me everything done in this bash process.

index=main sourcetype=journald source=Journald:Microsoft-Windows-Sysmon/Operational EventCode=1 PID=6788

Alright, looking at this process shows that the Parent CommandLine of this process was /bin/bash/preinstall.sh.

Correct! preinstall.sh

Complete

Oh duh, what does santa call you when you complete the challenge? whiz. Need to put that in my badge for the objective.

Unlocked hints after completing challenge lol.

Next: obj-10