Principal Technologist Middleware Mgmt. - (Observability & Cloud Monitoring)-90303802 -

Date: Sep 11, 2023

Location: US

Company: Amtrak

Your success is a train ride away!

As we move America’s workforce toward the future, Amtrak connects businesses and communities across the country. We employ more than 20,000 diverse, energetic professionals in a variety of career fields throughout the United States. The safety of our passengers, our employees, the public and our operating environment is our priority, and the success of our railroad is due to our employees.


Are you ready to join our team?

Our values of ‘Do the Right Thing, Excel Together and Put Customers First’ are at the heart of what matters most to us, and our Core Capabilities, ‘Building Trust, Accountability, Effective Communication, Customer Focus, and Proactive Safety & Security’ are what every employee needs to know and do to be most impactful at Amtrak. By living the Amtrak values, focusing on our capabilities, and actively embracing and fostering diverse ideas, backgrounds, and perspectives, together we will honor our past and make Amtrak a company of the future.


The Principal Technologist (Cloud Monitoring Portfolio) implements End-to-End Observability from ingesting metrics, logs, and APM, to configuring dashboards and alerting solutions for Amtrak DT Applications and provides visibility and insights into all tiers and components deployed in a multi-tier hybrid IT environment comprising IaaS(AWS, Azure), PaaS (Mulesoft, Salesforce, etc.,), SaaS, and on-premises data center. Also provides operations and maintenance (O&M) support for end‐users, applications, infrastructure, platforms, or network services, serving as an escalation point for Cloud Monitoring portfolio. As an Observability engineer, provides technical subject matter expertise, system design, administration, and technical support in a consultative manner. Works on tasks and projects or supports issues that involve a moderate to high degree of risk or may impact enterprise‐wide performance and makes use of the individual's high level of knowledge within one or more areas of specialty.



  • Design and implement end-to-end observability services for Amtrak DT using ELK stack and AWS MSK.
  • Manage the health of ELK and AWS MSK platforms.
  • Collect Logs, Metrics and APM data and User Experience data from Amtrak DT sources.
  • Implement index lifecycle management (ILM) using hot-warm-cold-frozen architecture to automate data streams management in a cost-effective manner. Backup data streams, create and manage searchable snapshots.
  • Leverage Kibana to create preconfigured, and editable dashboards, processing different sources of data and provide unified view of observability data.
  • Use time series modeling to detect anomalies in the current data and forecast trends and configure alerts to notify support teams.
  • Implement ingest pipelines for converting, enriching and processing the data as required. Process unstructured data using Elasticsearch ingest nodes and structure the data before indexing it into Elasticsearch.
  • Implement Real User Monitoring(RUM) and Synthetic Transaction Monitoring
  • Configure to extract new insights from observability data using AI OPs and machine learning features.
  • Evaluation of monitoring tools and solutions in Enterprise Observability such as Datadog, New Relic and Dynatrace. Assists and/or may lead in the development and analysis of source selection strategies.
  • This role requires working outside normal business hours on occasion.



  • Bachelor’s degree in computer science, engineering or equivalent plus 7+ years of relevant work experience, preferably with substantial experience in a project leadership or Subject Matter Expert capacity, required to satisfy education and experience requirements.
  • Must possess experience in Enterprise Observability platform tools and technologies with mid-expert skill level:
      • Elastic, Logstash & Kibana - (ELK stack)
      • AWS Managed Kafka or Confluent Cloud
      • AppDynamics or Datadog or Dynatrace or New Relic
      • Azure Monitoring
      • AWS Cloud Platform



  • Master’s degree in engineering, Computer Science or equivalent.
  • Working experience utilizing the ITIL Framework for Incident, Change, Problem, and Release Management.



  • Work is performed remotely.
  • May require travel up to 20%  

Must have excellent oral and written communication skills.


The salary/hourly range is $116,750 - $151,300. Pay is based on several factors including but not limited to education, work experience, certifications, internal equity, etc. Depending on an employee’s assigned worksite or location, Amtrak may consider a geo-pay differential to be applied to the employee’s base salary. Amtrak may offer additional incentive and pay programs to recognize and reward our employees, including a short-term incentive bonus based upon factors such as individual and company performance that is commensurate with the level of the position and/or long-term incentive plan compensation. In addition to your salary, Amtrak offers a comprehensive benefit package that includes health, dental, and vision plans; health savings accounts; wellness programs; flexible spending accounts; 401K retirement plan with employer match; life insurance; short and long term disability insurance; paid time off; back-up care; adoption assistance; surrogacy assistance; reimbursement of education expenses; Public Service Loan Forgiveness eligibility; Railroad Retirement sickness and retirement benefits; and rail pass privileges. Learn more about our benefits offerings here.​

Requisition ID:159656
Posting Location(s):N/A
Job Family/Function:Information Technology 
Relocation Offered:No 
Travel Requirements:Up to 25% 

You power our progress through your performance.

We want your work at Amtrak to be more than a job. We want your career at Amtrak to be a fulfilling experience where you find challenging work, rewarding opportunities, respect among colleagues, and attractive compensation. Amtrak maintains a culture that values high performance and recognizes individual employee contributions. 

Amtrak is committed to a safe workplace free of drugs and alcohol. All Amtrak positions requires a pre-employment background check that includes prior employment verification, a criminal history check and a pre-employment drug screen. 

Candidates who test positive for marijuana will be disqualified, regardless of any state or local statute, ordinance, regulation, or other law that legalizes or decriminalizes the use or possession of marijuana, whether for medical, recreational, or other use. Amtrak's pre-employment drug testing program is administered in accordance with DOT regulations and applicable law.  

In accordance with DOT regulations (49 CFR § 40.25), Amtrak is required to obtain prior drug and alcohol testing records for applicants/employees intending to perform safety-sensitive duties for covered Department of Transportation positions. If an applicant/employee refuses to provide written consent for Amtrak to obtain these records, the individual will not be permitted to perform safety-sensitive functions.


In accordance with federal law governing security checks of covered individuals for providers of public transportation (Title 6 U.S.C. §1143), Amtrak is required to screen applicants for any permanent or interim disqualifying criminal offenses. 

Note that any education requirement listed above may be deemed satisfied if you have an equivalent combination of education, training and experience.

Amtrak is an EOE/Affirmative Action Minority/Female employer, and we welcome all to apply. We consider candidates regardless of race/color, religion, sex (including pregnancy, childbirth and related conditions), national origin/ethnicity, age, disability (intellectual, mental and physical), veteran status, marital status, ancestry, sexual orientation, gender identity and gender expression, genetic information, citizenship or any other personal characteristics protected by law.