Akamai Splunk SIEM Connector On Kubernetes

A client who uses Akamai for their edge security wanted to download logs and events into Splunk for analysis and machine learning as they built new data models.

Akamai offers a SIEM Splunk connector

akamai/siem-splunk-connector

It was supposed to be easy to setup, and it was, after we figured out how to get it installed on a stand alone Splunk instance on our self-managed Kubernetes cluster :)

We deployed a stand alone Splunk instance and that was simple and easy, but when we configured the connector and enabled it all we got was errors inside of Splunk and no imported data.

The first challenge was identifying that this connector does not support OpenJDK…I had to use JRE.

That challenge was resolved but we were still getting errors:

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Access Token validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Access Token validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Client Secret validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Client Token validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Final Epoch Time validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Hostname validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Initial Epoch Time validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Limit validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Log Level validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Proxy Host/Port validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Begin Security Configuration ID(s) validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Client Secret validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Client Token validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: construct stanza…

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: done validation

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Error Checking complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Final Epoch Time validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: final_epoch_time=

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: get password service…

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Hostname validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: In validateInput, begin validate input

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: In validateInput, log_level=DEBUG

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: In validateInput, Service connect to TA-Akamai_SIEM app

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: In validateInput, stanza name = DLP

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Initial Epoch Time validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: initial_epoch_time=

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Limit validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: limit=

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Log Level validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: password validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Proxy Host/port validation complete

· <stderr> Argument validation for scheme=TA-Akamai_SIEM: Security Configuration ID(s) validation complete

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at com.akamai.siem.Main.main(Main.java:116)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at com.akamai.siem.Main.streamEvents(Main.java:344)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at com.splunk.modularinput.Script.run(Script.java:48)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at com.splunk.modularinput.Script.run(Script.java:74)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at java.util.concurrent.Executors.newFixedThreadPool(Executors.java:89)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1202)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” at java.util.concurrent.ThreadPoolExecutor.<init>(ThreadPoolExecutor.java:1314)

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” infoMsg = streamEvents, begin streamEvents

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” infoMsg = streamEvents, end streamEvents

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” javax.xml.stream.XMLStreamException: No element was found to write: java.lang.ArrayIndexOutOfBoundsException: -1

· message from “/opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh” Message : null, Exception : java.lang.IllegalArgumentException

· New scheduled exec process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py

· New scheduled exec process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_instrumentation/bin/on_splunk_start.py

· New scheduled exec process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py

· New scheduled exec process: /opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py

· New scheduled exec process: /opt/splunk/bin/splunkd instrument-resource-usage

· New scheduled exec process: /opt/splunk/etc/apps/TA-Akamai_SIEM/linux_x86_64/bin/TA-Akamai_SIEM.sh

After much head scratching and Googling, we finally identified that a max CPU was not set for the K8s pod…Once we set a MAX CPU setting in the kubernetes pod, the connector came up right away.

Here is the working deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: splunk-akamai-pod
name: splunk-akamai-pod
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: splunk-akamai-pod
template:
metadata:
labels:
app: splunk-akamai-pod
spec:

containers:
- image: splunk-jre:latest
resources:
requests:
cpu: “1”
limits:
cpu: “2”
imagePullPolicy: “Always”
name: splunk-akamai-pod

env:
- name: SPLUNK_START_ARGS
value: “ — accept-license”

- name: SPLUNK_PASSWORD
value: “mysupersecretpassword”

ports:
- containerPort: 8000

# Just sleep forever
command: [ “sleep” ]
args: [ “infinity” ]

Here is the service yaml

kind: Service
apiVersion: v1
metadata:
name: splunk-akamai
spec:
selector:
app: splunk-akamai-pod
ports:
- protocol: TCP
port: 8000
nodePort: 31163
type: NodePort

I hope this helps someone else experiencing the same challenges.