Offer summary

Qualifications:

Bachelor’s degree in Computer Science, Information Systems, or equivalent experience., 5+ years managing cloud-based lab/test environments, preferably in hybrid or multi-cloud settings., Proficiency in AWS, GCP, Azure, and IaC tools like Terraform and Ansible., Strong experience with observability stacks, especially Splunk, and scripting in Python, Bash, or PowerShell..

Key responsibilities:

Design, build, and maintain dynamic lab environments across AWS, GCP, and Azure for observability and security testing.

Collaborate with technical marketing and product engineering to define infrastructure requirements.

Automate provisioning and teardown of cloud lab resources using various tools.

Create operational scenarios to emulate common IT issues and ensure demo-ready infrastructure.

Job description

Description

The Observability Lab Engineer will be responsible for creating and maintaining Splunk IT observability solutions in the Splunk Center of Excellence (COE) lab. This role involves installation and configuration of Splunk tools that accurately reflect real-world conditions to support testing, development, and analysis efforts. The engineer will primarily collaborate with the Splunk technical marketing team to install, configure and maintain solutions in the COE lab for use creating demo content and “day in the live” situations to help share the power and value of Splunk with long time users, potential customers, industry analysts, and partners.

Key Responsibilities

Design, build, and maintain dynamic lab environments across AWS, GCP, and Azure to support observability, APM, and security testing.
Work closely with technical marketing, product engineering, and QA to define infrastructure requirements that reflect real customer deployments.
Automate provisioning and teardown of cloud lab resources using Terraform, Ansible, or cloud-native tools (e.g., CloudFormation, Deployment Manager).
Implement observability and monitoring pipelines (e.g., Splunk, OpenTelemetry, Prometheus) to simulate production-level visibility into test environments.
Create “day in the life” operational scenarios to emulate common IT issues, incidents, or performance degradation.
Ensure high availability, cost efficiency, and version control of cloud environments via scripting and automation.
Maintain documentation for lab environment setup, automation scripts, and testbed configurations.
Provide support during live demos, webinars, and product showcases with reliable, demo-ready infrastructure.

Minimum Qualifications

Bachelor’s degree in Computer Science, Information Systems, or equivalent professional experience.
5+ years of experience managing cloud-based lab/test environments, preferably in hybrid or multi-cloud settings.
Proficiency in cloud platforms (AWS, GCP, Azure) and IaC tools (Terraform, CloudFormation, Ansible).
Strong experience with observability stacks, especially Splunk, plus exposure to APM, log aggregation, and metrics systems.
Scripting experience in Python, Bash, or PowerShell to automate deployment, monitoring, and validation.
Experience with containerized environments (Kubernetes, Docker) and service mesh configurations.
Solid troubleshooting skills and knowledge of networking, security groups, and CI/CD pipelines.
Strong documentation and communication skills for cross-functional collaboration.
Comfortable supporting live demos and resolving environment-related blockers under pressure.
Experience with performance benchmarking tools and load generators (e.g., JMeter, Locust)
Familiarity with demo orchestration frameworks or event injection tools for synthetic data
Knowledge of GitOps practices and DevOps pipelines for continuous environment refresh
Splunk Core Certified User or Power User (or similar tool experience) - nice to have

Splunk, a Cisco company, is an Equal Opportunity Employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, gender, sexual orientation, national origin, genetic information, age, disability, veteran status, or any other legally protected basis.

Thank you for your interest in Splunk!

Required profile