Elastic Technical Deep Dive Interview PDF

Initial info: “During the Technical Deep Dive, we will discuss your background, experience and comfort with the following topics, which are commonly encountered in the day-to-day of a Solution Architect at Elastic. The topics are as follows: Observability ○ Monitoring today is not...

Initial info: “During the Technical Deep Dive, we will discuss your background, experience and comfort with the following topics, which are commonly encountered in the day-to-day of a Solution Architect at Elastic. The topics are as follows: Observability ○ Monitoring today is not just logging. We will be discussing the overall concept of Observability, how it fits today’s needs in terms of incident responses, overview of business critical processes, and other key requirements. Expect an observability conversation around APM, Logging, Metrics, Transactions and more. Security ○ Attacks are on the rise, and Security Information and Event Management (SIEM) systems along with EDR are key for proactively responding and protecting organizations. We will be talking about the security landscape along with the latest trends, mainly focused on EDR, EDP, SIEM, SOC and many other topics related to the role of a security analyst. Search ○ Developing applications requires a lot more than just developers, and providing a relevant search experience is more important than ever. We will discuss what it takes, including architectural and database elements, to create a search experience and why this is important in today’s world. We are always searching, and users are always searching. Cloud ○ Because the world is moving to the Cloud, merging all the technical concepts that we know with a Cloud focus is a MUST. Be ready to talk about the different Cloud Providers, architectures, components, migration strategies among others. Solution Architecture & Estimation ○ From estimating resources, to building comprehensive architectures, our role is to ensure the systems we architect are cohesive and meet the needs of our customers. We will focus on all the moving parts behind estimation and creating architectures with modern components such as Kubernetes. Chat with Bobby Notes: They say choose what you’re most comfortable with then they ask you 3 levels of questions, such as “how would you define observability?” - Level 1 Level 3 - “Tell me what a HA application design would look like?” [Typically Level 3 is catered around design and architecture questions. Elastic doesn’t design them but monitors them.] - It’s important to understand how those systems are designed. Answer what I know and say I don’t know. Don’t lie and try to answer everything. - Folks are typically really good at one, meh at another, and ok at the rest. - No set number of questions and trying to make it more conversational. - Security: - Research these terms and how they work together. Elastic makes a ton of money off SIEM - SIEM - Alerts come from every system. When cyber attacks became more often, all data and resources needed to be in one central repository. Started as log aggregation, then started ML and search on it. - Endpoint - Attacks usually start at the endpoint because humans make mistakes. - CrowdStrike is the biggest EDR vendor. They are a competitor but also a data source. Can keep using CrowdStrike and use Elastic for data search. - They will focus on the why and importance of things. - Matt is more security background but has always been on the sales side. Leans more security here but has to know the other stuff. - Observability: - Have an opinion of what observability means. Understand that term and compare it to what monitoring. Compare monitoring vs. observability. - Observabilility: The ability to observe an application for an uptime for behavior and performance. The goal is to understand when it wil lgo down. - Bring up MTTR, MTBF, calculating availability (what that means) - 3 main pillars: - Log data: Can see errors in logs - Metric data: How an application may be performing based on utilization, networking, etc… - Transaction data: Traces within the code itself. Understand where the code is spending the most time in the particular functions. - X-Ray is for observability.  - Search: - It is very broad. It is not super specific. Think about how search is used all the time. Talk about search expectations when using Google. Need relevant results and need them fast. Think about what a search experience is if even using SQL or some basic language for searching a database. - Splunk is biggest competitor: - Similar use cases around SIEM and observability. - But they don’t have any capabilities around search. - Level 3 could be something around a search design or system. GenAI and RAG can be talked about. - Cloud: - Basics - SA and Estimation: - Most customers in SLED don’t use Kubernetes. Some do but it’s important to understand estimating a solution. Think of an AWS service that you have to go off and selling the service, how do you size it according to size of the resources and pricing. - May ask a little bit about pricing. If estimate wrong, that ruins customer trust. Can give an example of what that looks like. - There will be some direct and some conversational. - Draw from my experience and do some studying on security. See that Elastic is a SIEM and EDR. Possible Q’s: Observability 1. Can you explain the concept of Observability and how it differs from traditional monitoring? Observability focuses on understanding the internal states of a system based on its outputs, like logs, metrics, and traces. Traditional monitoring, on the other hand, typically involves predefined checks to determine if a system is up or down. Observability enables proactive problem-solving by providing context about what is happening and why, making it essential for debugging complex distributed systems. 2. How would you design an Observability solution for a distributed microservices architecture? I’d design an Observability solution by implementing standardized logging, metrics, and distributed tracing across all services. Logs would include contextual details like trace IDs for correlation, metrics would track performance (e.g., latency, error rates), and tracing tools like OpenTelemetry would visualize end-to-end request flows. These outputs would feed into a central platform like Elastic Stack for aggregation, visualization, and actionable alerts. 3. What are the key components of APM (Application Performance Monitoring), and how do they contribute to troubleshooting performance issues? Key APM components include transaction tracing, service maps, error tracking, and performance metrics. Transaction tracing identifies bottlenecks in request flows, while service maps provide a high-level view of dependencies. Error tracking pinpoints failing components, and metrics like response time or throughput help quantify performance issues, enabling teams to pinpoint root causes efficiently. 4. Describe a scenario where you used logs, metrics, and traces together to diagnose a production issue. In a production issue involving increased latency, I started by reviewing metrics, which showed high request durations in one service. Logs provided more context, revealing repeated database query failures. Distributed traces confirmed the problematic query’s cascading effects across services, enabling us to optimize the query and resolve the latency. 5. What strategies would you implement to ensure Observability data is meaningful and actionable for both technical teams and business stakeholders? To ensure Observability data is actionable, I’d create dashboards tailored to technical metrics (e.g., error rates, CPU usage) and business KPIs (e.g., transactions per minute). I’d also define alert thresholds aligned with service level objectives (SLOs) and establish processes for analyzing trends to proactively address issues before they impact users. Security 6. Can you explain the role of SIEM systems in modern cybersecurity strategies? SIEM systems collect and analyze logs from various sources to detect, investigate, and respond to security threats. They centralize data to provide visibility into the security posture and use correlation rules and machine learning to identify patterns indicating malicious activity. 7. How would you design a security architecture to detect and respond to threats in real time for a public-sector client? I’d design a layered security architecture combining SIEM for centralized monitoring, EDR for endpoint protection, and a Security Orchestration, Automation, and Response (SOAR) system for real-time remediation. Integrating these tools into a SOC ensures continuous monitoring and swift incident responses. 8. What is the difference between EDR (Endpoint Detection and Response) and EDP (Endpoint Data Protection), and how do they complement each other? EDR focuses on detecting, investigating, and responding to endpoint threats, while EDP ensures the confidentiality and integrity of data on endpoints. Together, they offer comprehensive endpoint security, addressing both active threats and data protection requirements. 9. Describe the steps involved in implementing a Zero Trust security model. Implementing Zero Trust starts with strong IAM practices like multi-factor authentication and least privilege access. It includes micro-segmentation to isolate resources, continuous monitoring for anomalous behavior, and enforcing end-to-end encryption to secure data in transit and at rest. 10. How do you prioritize vulnerabilities identified during a security assessment? Provide an example of a critical vulnerability you mitigated. I prioritize vulnerabilities based on their impact and likelihood using frameworks like CVSS. For example, I once mitigated a critical vulnerability involving exposed administrative ports by restricting access to internal IP ranges and implementing MFA for administrative accounts. Search 11. What architectural considerations are important when designing a search experience for a large-scale application? Scalability, relevance, and performance are key. I’d use a distributed search engine like Elasticsearch with sharding for scalability, implement relevance tuning (e.g., boosting key fields), and optimize queries for performance using caching and filters. 12. How do you approach indexing and optimizing search for unstructured data (e.g., logs, documents)? I’d preprocess unstructured data to extract meaningful fields and tokenize text. During indexing, I’d apply appropriate analyzers (e.g., stemming) and configure mappings. Query optimizations like filtering on indexed fields improve performance, while relevance tuning ensures better search results. 13. What are the trade-offs between relevance, performance, and scalability in a search solution? Improving relevance (e.g., complex scoring functions) can increase computational overhead, affecting performance. Scaling horizontally with distributed nodes can address performance issues but adds complexity. Balancing these factors requires understanding user needs and adjusting resource allocation. 14. Can you describe how you would integrate search capabilities into a multi-tenant application? For multi-tenancy, I’d use index-level isolation (separate indices per tenant) or document-level tagging with access controls. This ensures data segregation while leveraging shared infrastructure. Properly scoped queries ensure tenants only access their data. 15. Explain the role of stemming and tokenization in a search pipeline. Stemming reduces words to their root forms (e.g., “running” to “run”), improving recall by matching variations of a term. Tokenization breaks text into smaller units, such as words or phrases, enabling efficient indexing and query matching. Cloud 16. Describe the key differences between IaaS, PaaS, and SaaS, and provide examples of when you would recommend each. IaaS provides infrastructure resources (e.g., VMs) for full control and flexibility, suitable for custom workloads. PaaS offers managed platforms for developers (e.g., serverless compute), ideal for rapid application development. SaaS delivers ready-to-use applications (e.g., email), best for end-user needs. 17. How do you approach a cloud migration for a legacy on-premise application with minimal downtime? I’d perform an initial assessment of dependencies and design a hybrid architecture to support data replication during migration. A phased approach with canary deployments ensures smooth transitions, with continuous monitoring to address issues promptly. 18. What are the trade-offs between a multi-cloud and hybrid-cloud strategy? Multi-cloud improves redundancy and avoids vendor lock-in but adds complexity in management and integration. Hybrid-cloud combines on-prem and cloud benefits but may involve higher costs and latency due to data movement. 19. Explain the shared responsibility model in cloud security and how it applies to public-sector clients. In the shared responsibility model, the cloud provider secures the infrastructure, while the client secures applications, data, and user access. For public-sector clients, this means implementing strong IAM and encryption to meet compliance requirements. 20. Can you describe the process of implementing auto-scaling in a cloud-based application and its challenges? Auto-scaling involves setting policies to dynamically adjust compute resources based on demand. Challenges include accurately predicting thresholds, avoiding over-provisioning, and ensuring stateful applications can scale seamlessly without disrupting services. Solution Architecture & Estimation 21. How do you estimate resource requirements for a large-scale system involving multiple integrations? I’d analyze historical data, conduct load testing, and consider integration points’ resource needs. I’d also include buffers for peak loads and factor in scaling policies to accommodate unexpected surges. 22. What are the key considerations when architecting a solution that uses Kubernetes for container orchestration? Considerations include defining pod resource limits to prevent resource contention, using namespaces for isolation, and implementing ingress controllers for traffic management. Monitoring and securing the Kubernetes cluster are also critical. 23. Describe a time when you had to balance competing technical and business requirements to design an architecture. In a project requiring high performance and cost control, I used serverless for variable workloads and reserved instances for predictable tasks. This met business cost goals while ensuring technical scalability and performance. 24. How do you ensure that a solution architecture remains scalable and cost-effective over time? Regular architecture reviews and monitoring resource usage ensure scalability. Cost-effectiveness is achieved by leveraging reserved instances, optimizing workloads, and automating infrastructure management with tools like Terraform. 25. Provide an example of a solution you designed from scratch. What was the biggest challenge, and how did you overcome it? I designed a multi-region, fault-tolerant web application using load balancers, replicated databases, and auto-scaling groups. The biggest challenge was ensuring data consistency across regions, which I resolved by implementing conflict-free replicated data types (CRDTs) for eventual consistency.  Possible Q’s: Level 1: Basic Knowledge Level 2: Knows the Value Level 3: Practitioner Observability 1. Are you familiar with the term 1. Can you describe what an 1. Imagine you have “observability”? If not, how about incident management process inherited a large web app monitoring (logging, metrics)? looks like? (100+ servers) that just had major unplanned downtime 2. How would you compare 2. Can you describe what the which impacted business logging data and APM data? How competitive landscape is for operations at your company. about logging and metrics? observability? Where do you Your first task is to make see Elastic filling in? sure that there aren’t any 3. Why is real-time visibility for more surprises. Describe logs and system metrics 3. How do you define your overall strategy. important? success for observability (MTTD, SLAs, etc…) 2. Imagine you have a cloud-based-web app and you are currently able to view logs through a near-real time process. Your charts stopped showing new incoming data 30 minutes ago. How would you troubleshoot? Security 1. Why do customers need SIEM, 1. Can you describe what an 1. Talk through a recent threat hunting, and endpoint incident management process public security breach and security tools? Differences looks like in a SOC? explain why it happened, between these tools? consequences of a breach 2. What are the new trends in like you are describing, and 2. What is the the security industry (XDR, how it could have been difference/similarities between SOAR, etc…)? avoided. compliance and security? 3. Can you describe what the 2. Imagine you just competitive landscape is for uncovered a ransomware security? Where do you see incident. Describe how it Elastic fitting in? likely happened, what steps you would have taken to detect the incident and how you would begin to remediate. Search 1. What are some use cases 1. What are some typical 1. A lot of databases have enabled by Search? Different customer requirements for full-text search capabilities or market verticals. search applications? achieve similar functionality (answers - speed, scale, with features such as  2. Give a few examples of relevance, latency, etc…) secondary indices. Why different types of search would you use a search experiences? What kind of 2. Once you deliver a search system? flexibility is enabled by Search? application to a customer, what types of analytics are 2. You get no response to a valuable to ensure your search query that has been customers get the best issued successfully multiple experience? times in the past? What do you do? 3. Who are the teams that are typically involved with 3. How do you scale a search customer facing system? What are some ways search-engine applications of scaling a search system? (answer - engineering, dev, and marketing). How do you work with change management for all teams to be successful? Cloud 1. What is your favorite cloud 1. What is the different 1. Imagine you have a provider? Why? between cloud providers, portfolio of 100 on-prem regions, and zones? What do apps that you are responsible 2. What is the value of cloud? you need to consider for for. You are asked to select a choosing to run your list of candidates for a application? migration to the cloud. How do you make the list? What 2. What is your preferred are the most important way to run a cloud factors you need to consider? application - IaaS, PaaS, or SaaS? If all choices were 2. If you had to help a available, how would you customer save 25% on choose between them? overall cloud spend in the next 12 months, how would 3. Why would you choose a you do it? multi-cloud strategy? Solutions 1. How have you moved a TB of 1. Have you ever paid for 1. It’s black friday and you Architect data from one system to another? software that was open need to prepare to 10x your source or free? What architecture. What are some convinced you to actually strategies you could use to open your wallet? scale it? 2. You are assigned a task to move 10 apps of varying sizes from one data center to another. How? Observability Level 1: Basic Knowledge 1. Are you familiar with the term “observability”? If not, how about monitoring (logging, metrics)? Observability is an investigative approach. It looks closely at all system components and data collected to find the root cause of issues. It involves things like trace path analysis which follows the path of a request to identify integration failures. Observability is all about looking at systems from a wider view with historical data and system interactions. It really helps with alerts and getting notified about what’s happening. Observability (collect) → monitor [visualize and create dashboards] → analyze [see if criteria meet SLOs and SLIs] Observability is the why and how. Monitoring is the when and the what. Observability for anomalies helps further investigate them even if they occur because of the interactions between different service components. Monitoring helps you discover those anomalies. For cause and effect, monitoring will measure the values to see if there is an effect and observability finds the cause. If we’re releasing code, monitoring tracks the system metrics for things like load times and data retrieval then observability finds the reason. Monitoring is a must for proactive error-catching because it helps you identify and fix issues before they cause long-term consequences. With microservices, observability is really important because it helps identify bottlenecks for things like distributed tracing, health checks, log aggregation, app metrics, and auditing. - Logs describe the specific events of what happened and when - Metrics are usually numeric and used for things like KPIs, timestamps, and names, that help provide context to the logs - Traces are the mapped journey of a given request as it moves through the system. Tracking it through microservices helps find bottlenecks. Summary: Observability: understanding internal state of systems focused on proactive debugging and why/how Monitoring: Focus on predefined checks and metrics for system health to see if a system is up or down, the what. APM: Tracking performance of application like response times, error rates, and transaction tracing to find bottlenecks. 2. How would you compare logging data and APM data? How about logging and metrics? Application Performance Monitoring (APM) monitors common metrics like cpu usage, response times, error rates, transaction traces, and instances. It helps ensure business-critical apps are maintained where APMs can give real-time data and insights into the performance of apps. Typically, IT, DevOps, and SREs are using APMs to pinpoint issues. It’s good for rapid diagnosis and customer satisfaction APM is for performance while log is for data analysis and compliance. They can overlap at times and actually build a clear observability case. Logs help with events to find application errors and remediation and track user activity while APM is more focused on identifying is a server needs more resources, identifying peak seasonal traffic, and finding external issues that can be causing performance degradation. Summary: Logs provide detailed event data for deep investigation and shows exactly where things may have failed APMs focus on the performance and behavior of applications to show things like response times, service dependencies, and giving broader insights. 3. Why is real-time visibility for logs and system metrics important? Real-time visibility for logs and system metrics is important because it gives insights into what is happening at that exact moment. In situations where you’re dealing with highly sensitive environments, whether that’s 911 systems, public safety resources, or even general applications, it’s important to pin-point where things go wrong. By having millisecond logs and metrics, a map can be made of what happened, why, where, when, and the impact it had on the general system. It really helps for faster incident response to pinpoint the root cause of issues which reduces the mean time to discovery (MTTD), which increases your overall availability. These real-time insights in combination with alarms helps a team know about a problem as it’s happening rather than after the fact. Level 2: Knows the Value 1. Can you describe what an incident management process looks like? Incident management is how a team responds to an unexpected impairment. That could be something like a loss of network connectivity, an availability zone outage, or even a security incident. The whole goal is to get back to regular operations and continue delivering business value as soon as possible. It’s also important to do some chaos tests in controlled environments here and there to understand what may happen or how your systems may react. The overall process for incident management goes like this: 1. Identify the risk: First the most important resources are identified. For example, a company like Amazon’s biggest risk is that checkout button. It doesn’t matter if recommendations are working and you can search. If you can’t click checkout, Amazon can’t make money. 2. Protect assets: Strengthen the security of those critical resources. 3. Detect incidents: Setup monitoring to detect these incidents before a customer does. 4. Respond to incidents: Either stop the incident or contain it. This can be something like spin up the backups, redirect traffic, or setting up automation to scale away from the issue. 5. Recover: Set up a root cause analysis to figure out what happened, capture lessons, and build a better recovery plan if necessary. 2. Can you describe what the competitive landscape is for observability? Where do you see Elastic filling in? I’m not too familiar with the entire industry, but I know that the big names are Datadog, Splunk, and Elastic. I know Datadog is big with cloud-native apps and Splunk is good for enterprise systems but expensive. One I’ve heard of but don’t know much about is Grafana. My customers sometimes mention they use Grafana dashboards but I haven’t used it. I’m familiar with Elastic because of OpenSearch on AWS. Most of my customers use it because it’s good for easily searching through logs, metrics, and even for ML. They use it a lot for ML vector searches. - Elastic is known for being open source and working across hybrid/multi-cloud environments. It’s really unified and lets users have logs, metrics, and advanced search in one place. - I know the biggest selling point is the unification and the search. Having everything together, being able to quickly search through all logs and metrics, then using ML on those logs to determine if something might happen in the future or optimize resources. 3. How do you define success for observability (MTTD, SLAs, etc…) I like to think about some of the availability metrics. The ones that come to mind are things like mean time to recovery (MTTR), mean time to detection (MTTD), and mean time before failure (MTBF). These values can be used to find availability and calculate reliability. Reduce MTTR: Automatically failover and have runbooks, use things like containers instead of basic instances. Reduce MTTD: Use proactive monitoring and have more granular healthchecks. Increase MTBF: Find bugs in dev, use chaos engineering, minimize the blast radius of failures, and deploy smaller changes. I also know about service level agreements (SLAs) and looking at latency. With SLAs, there’s a value customers are given where if they’re not met, they’ll get some money back. That’s why observability and calculating it is really important, it can hinder customers and the business itself. When I talk to government customers, there’s always a requirement for a disaster recovery plan, a set recovery time objective (RTO), and set recovery point objective (RPO). Those can be calculated based on how long it takes another environment to spin, how many availability zones an application is in, and which resources are truly mission-critical. Level 3: Practitioner 1. Imagine you have inherited a large web app (100+ servers) that just had major unplanned downtime which impacted business operations at your company. Your first task is to make sure that there aren’t any more surprises. Describe your overall strategy. The first thing I’d do is a root cause analysis. Figure out what went wrong and how it could’ve been mitigated. Once that analysis is set, I’d set up better observability and monitoring. The monitoring metrics could be more granular so I can see what’s happening then use observability to find the why/how. With these better systems, I’d set up alarms that trigger at smaller scales. For example, instead of triggering an alarm when latency grows by 15, it’d be set to 10. Then I’d create a few central dashboards that everyone can monitor at all times. It’d have all key business resources available and show all pieces of the environment. I’d want to set up automated run books and redundancy after that. There’s no reason an engineer should manually click anything if there’s an impairment because what if they’re unreachable or the incident is causing internal systems to slow down. The redundancy makes sure that there’s a backup just in case this happens again. Last but not least, I’d do chaos engineering tests. Why should we figure out what happens during the impairment? I’d run tests on environments to see how our teams react, which parts of the systems would be impaired, then create a better plan for everyone and everything involved. 2. Imagine you have a cloud-based-web app and you are currently able to view logs through a near-real time process. Your charts stopped showing new incoming data 30 minutes ago. How would you troubleshoot? The first thing I’d do is confirm if it’s a chart problem or an application problem. I’d look at logs and metrics past the dashboard to confirm that data is still flowing in and the dashboards aren’t the only thing going wrong. After that, I’d check metrics and review all the logs to see where things started going wrong. Did a lot of requests come in from a single IP right before the dashboards cut off, did CPU utilization scale up or did networking slow down, or was a push made just 30 minutes before? From there, I’d try to replicate the issue by generating more logs and seeing if anything comes in. I’d restart the failed systems, clear up any of the bottlenecks if there was an influx of requests, or talk to the rest of the team like SREs to see if there was anything happening on their end. Security Level 1: Basic Knowledge 1. Why do customers need SIEM, threat hunting, and endpoint security tools? Differences between these tools? Security Information and Event Management (SIEM) collects logs and events for further analysis such as dashboards, visualizations, alerts, and searches. Some SIEMs will automatically remediate threats in a system and find other ones that may be hard to typically identify. SIEM collects data from every piece of an ecosystem whether it’s apps, devices, networks, and allows it all to be analyzed from one central hub. They’re mainly security pieces and is good for things like continuous monitoring, log management, and changes in user actions. Threat hunting is a way to find security incidents and threats in a system that automated detection may have missed. Those incidents can be found using manual or automated tools that identify suspicious behavior. It finds things like visibiltiy gaps, data collection issues, and then figure out how to automatically detect it in the future. There are multiple types of hunting types: - Hypothesis driven hunts: Hunting based on potential threats and ideas then confirming or denying it with data. - Baseline hunts: Create a baseline for normal procedures then identify the outliers. - Model-Assisted Threat Hunts (MATH): Using ML to create models of good/bad behavior then putting it up against recent activity. Endpoint security tools: Endpoint security tools protect user devices like phones and computers from malicious software. Endpoint security tools prevent bad actors from getting into corprorate networks and enterprise systems. It prevents things like phishing, ransomware, and internal seucirt risks which is good for reduced response times, compliance, and general awareness. Some tools let you block unkown threats and focus on the attacks sometimes using AI to automatically respond and assist. They typically capture data across multiple views to visualize the data and respond with ML. Summarize: SIEM: Security Information and Event Management combines all data from every location into a central location. Threat Hunting: Find the threats yourself either through automation or manually to find possible holes in security. Endpoint security: Protect users from unknown threats on their own devices. 2. What is the difference/similarities between compliance and security? Compliance: Compliance is regulated pieces of your application to meet standards that the regulation sets. For example, to meet HIPAA compliance, PII needs to be covered, logging needs to be enabled, and encryption has to be on at all points. Security: Security is making sure your resources are inaccessible from the wrong people. It keeps your application running, private, and protected from threats. This can be anything from malware to human error. In most cases, you have to be secure to be compliant. Level 2: Knows the Value 1. Can you describe what an incident management process looks like in a SOC? SOC stands for Security Operations Center - the whole goal of incident management is to get to the bottom of the situation. It’s important to know what happened, how, who, why, and how to avoid it in the future. 1. Identify: The first thing to do is identify the incident. That can be through monitoring, detection systems, or even reports from employees. 2. Asses: The incident needs to be assessed for its scale and impact. This helps determine how important it is. 3. Contain: The incident needs to be contained before it spreads across systems. 4. Investigate: Now that it’s contained, an investigation needs to be done to find the root cause. Doing things like look at log or see which specific systems were impacted. 5. Respond and Remediate: With the incident understood, the response plan is made. That can be things like patching, restoring data, getting rid of malware, or even adding more security measures. 6. Communicate and Reporting: Everything that’s done is written down and reported to the greater team/stakeholders. 7. Post-incident analysis: De-brief and understand the effectiveness of the incident response. It’s good to analyze what was done and if it was fast, effective, and can prevent incidents in the future. 2. What are the new trends in the security industry (XDR, SOAR, etc…)? XDR: Extended detection response: One step past endpoint detection response and SIEM which pulls all security data into one central location for advanced analytics typically through AI/ML. SOAR: Security, Orchestration, Automation, and Response: Unify security tools to integrate with a SOC and respond to threats. They use things like APIs, plugins, and custom integrations to connect tools and coordinate activities. Connects to things like Jira, slack, and emails. An example is patching to monitor and automatically apply patches to reduce manual monitoring and updating. It can also be used for malware detection on an endpoint then immediately check if other endpoints have the same vulnerability. The automated response can quarantine and isolate the infected hosts before it spreads. 3. Can you describe what the competitive landscape is for security? Where do you see Elastic fitting in? There are a good number of security specific companies like Palo Alto Networks and CrowdStrike playing with security but there’s also other players like AWS and IBM that have security related services. Based on what I’ve heard with my customers, they almost always use a cloud provider + a security vendor that they find easy to integrate with. A common one I typically see is fortinet but I’m not familiar with their services. I know Splunk is one of Elastic’s biggest competitors but to be frank, I haven’t used them and don’t know their value proposition besides being known as one of the most popular options. Elastic itself has a unified platform for SIEM, threat hunting, and XDR. It’s also available on AWS GovCloud and is FedRAMP moderate compliant. It’s open source and a lot more affordable compared to Splunk but from what I’ve read, the real power comes from the flexibility with the search and dashboards. I’ve used some other databases in general to query data but when I used vector searches with opensearch, it was much faster. Since ML and security are rapidly growing, a lot more customers are going to have a lot more data and the slower, expensive systems aren’t going to cut it. It also helps to cut everything down to one single provider rather than being separated across multiple systems. But, even if you are, I know that Elastic connects with a lot of different resources to easily integrate and scale with other platforms. Level 3: Practitioner 1. Talk through a recent public security breach and explain why it happened, consequences of a breach like you are describing, and how it could have been avoided. The one that always comes to mind, since I work at AWS, is the Capital One data breach. In 2019, there was an exploit on a web application firewall (WAF) that gave attackers access to over 100 million customer accounts and credit card applications stored in S3. They storeed things like names, addresses, SSNs, and credit scores. What happened was there were misconfigured AWS WAF rules that allowed the intruders to list and access the S3 buckets. The IAM roles weren’t very strict, there were no real-time monitoring and detection alerts, and the attackers used a pretty common exploit on a known vulnerability to get in. It could’ve been avoided with proper WAF permissions and least-privelege access across the board for users. They could’ve also used a real-time monitoring dashboard or set up alerts to find things like this. Had they used a service like Elastic that works on top of their cloud environment it could’ve been avoided. I think with some ML and AI tools that scan across their entire environment, it could’ve found that open S3 bucket. Last but not least, they could’ve had something like a SOC or XDR team that’s actively investigating problems and mitigating them before they scale. Had they done things like threat hunting where they’re proactively searching for vulnerabilities or chaos tests that simulate these types of events, it could’ve been avoided entirely. 2. Imagine you just uncovered a ransomware incident. Describe how it likely happened, what steps you would have taken to detect the incident and how you would begin to remediate. If there is a ransomware attack, it probably happened from a phishing email, open ports/leaked access keys, or exploiting an unpatched application. After that, they got access to more critical systems then looked for easy targets that house data like critical databases or backups. To detect the incident, I would have set up SIEM alerts to flag any suspicious activity like any unexpected access or IPs from out of the country. I’d check for RDP access, powershell commands, or unauthorized changes. I’d also look at EDR to identify any malicious points on hardware and build out SOAR within a SOC so it could automatically be mitigated. I’d remediate by isolating the ransomware and prevent it from spreading across the rest of the system then reset all passwords, decommission compromised accounts, and enforce zero-trust/least-privelege. Then I’d asses the scope fo the damage by looking at logs and alerts to understand where it all came from. Next, I’d patch and update systems to close out the vulnerabilities. After that, notify everyone that was impacted and stakeholders reminding them that it has been mitigated but security is the utmost importance. And to finish, I’d do a post incident review to see how it started, what went wrong, why, and build out next steps to prevent it from happening again. Search Level 1: Basic Knowledge 1. What are some use cases enabled by Search? Different market verticals. In my experience in the public sector, I see customers using search to quickly look through large data sets that are filled with small files for quick searches. The one that came up most often was IoT sensor data that was then mapped on a graph for geolocation and the ability to search through all this data so quickly was important since the sensors were used in emergency response systems. Some other customers used it for justice and public safety court documents and records then I’ve seen it used in ML for vector search since data sets are typically really large. One customer used it to rank natural language searches to give the best results when searching for hotels/apartments across the US. 2. Give a few examples of different types of search experiences? What kind of flexibility is enabled by Search? There’s things like semantic search, predictive search, and multimedia search. Semantic search is using natural language to do queries like Google or ChatGPT, then there’s predictive on things like Amazon.com to autofill what you’re typing, and multimedia to search with images. There’s a few more but all of them make search experiences more enjoyable, accessible, efficient, and personalized for the user. It allows for unified searches across different data sources and even search ranking/learning. For example, if I search for headsets and microphones, I’ll get more gamer focused responses since I play some video games. It also helps by finding real-time results within data stores without having to use filtered searches, something typically seen in hotels or the automobile industry. In security, these searches make it easier to sift through really large datasets in quick speeds without hindering the user experience. Level 2: Knows the Value 1. What are some typical customer requirements for search applications? (answers - speed, scale, relevance, latency, etc…) The most common ones are speed and low latency. When a user is on Amazon.com or Netflix, if it takes long for the results to pop up, the user gets frustrated. For websites with more competition, like Walmart vs. Target for example, whichever application loads faster wins the customer. The next is scalability. To be able to get the right results from your search, there needs to be enough data. It can’t only work well with a small set of resources. The application has to be able to scale with its users. Next, it needs customizability with things like rating and proximity. If I search for Indian food, I don’t care about restaurants in New York if I’m in NC. It’s a major need and most times, users want to see what users around them are interested in, whether that’s food, shopping, events, or anything else. In public sector specifically, search needs relevance. A lot of my customers in justice and public safety are looking through hundreds of thousands of files, videos, images, and evidence resources. They need to be able to find what they’re looking for and getting access to those resources fast is important, especially in a crisis response. 2. Once you deliver a search application to a customer, what types of analytics are valuable to ensure your customers get the best experience? I’d look at the search query analytics like popular topics or gaps in the content. I’d look for searches that return no results and see how people are searching, specifically looking at which terms they’re using and how the search application can be modified. I’d then look at click through rate to see how often users are clicking on search results. I help my friend’s dealership and run the website hosting/security/analytics and something we always look for it bounce rate. If they only visit the page once then leave, that may be an indication of results not being good enough or too far down on the page. There are other things like how long a user spends interacting with the search results or abandonment rate, which is similar to the bounce rate. I’d then look at the performance side focusing on query latency to see how long it takes for results to pop up then scalability and load. If there’s slower performance during peak usage, then I’d look at ways to improve scale in general. 3. Who are the teams that are typically involved with customer facing search-engine applications (answer - engineering, dev, and marketing). How do you work with change management for all teams to be successful? For customer facing search apps, the teams involved are engineering/dev teams, marketing, and data analysts. The engineers are deploying, maintaining, building, and fine-tuning the search. They’re doing things like updating the algorithms, improving scale, and enhancing the app to improve performance. Marketing is improving the content itself and fixing the discoverability/relevance of products. When a user uses the search, it doesn’t make sense to show results that aren’t a good indication of what they’re looking for. They’d be improving images and fixing result names to appear based on what users are typically looking for, Then there are data analysts. They’re looking at what users are searching for, measuring the performance itself, seeing how long users are staying on application and how many times they click results. Their job is to identify relevance through the insights and report the user trends to the devs and marketing. For change management, it’s important to have cross functional collaboration and clear communication. All teams need to have somewhat regular syncs to align on goals and timelines where they can use shared tools like Jira to make requests to one another. With clear communication, they can clearly define the metrics teams are looking for and work backwards based on user feedback. This prevents any teams from being silo’d and ensures everyone is on the same page. - On the technical side, there’s a need for versioning and documentation. With versioning, there are clear phased approaches with testing, validation, and performance tuning. This allows for things like a/b testing and consistent updates for feature changes and system upgrade. With documentation, there can be training for all teams available to improve the onboarding and debugging experience. I always tell people to leave a paper trail because if someone has to come in after you and fix something, there’s a clear path to do so. Level 3: Practitioner 1. A lot of databases have full-text search capabilities or achieve similar functionality with features such as secondary indices. Why would you use a search system? When I look at comparing a search system vs. a database, the first thing I think about is structure. A database usually has structured data that is organized and designed for backend systems. A search system is for throwing in all kinds of information that can be unstructured with a focus on users directly querying from it. Search systems are better fo high-speed queries, full-text search, customizable relevance, and native tools for analyzation. On the other hand, databases don’t require duplication, support ACID transactions, can be easier to implement, and are sometimes even more cost effective. For systems like e-commerce, log analytics, and geospatial, search systems are stronger. For simpler use cases where cost, simplicity, and a structured backend are necessary, databases are better. 2. You get no response to a search query that has been issued successfully multiple times in the past? What do you do? 1. I’d verify the query is correct and that the syntax hasn’t changed 2. Then I’d confirm the data itself exists by running a similar query to make sure it hasn’t been deleted or modified. 3. Then I’d look at system logs for errors, timeouts, or anything out of the ordinary. 4. After that, I’d check the system itself for infrastructure changes like updates, any node failures, insufficient resources, or anything related to the application observability. 5. Next, I’d confirm connectivity and ensure that everything is connected properly. I’d do route tracing, check API keys, access control, and networking. 6. Then I’d reindex the data to make sure that everything pops up 7. Finally, I’d monitor the changes and set up alerts to change anything like this from happening again. 3. How do you scale a search system? What are some ways of scaling a search system? There are a few ways to scale a search system. The first and most common one is scaling up by adding more powerful hardware and resources. It’s costly but works and is simple. The next is horizontal scaling by adding more nodes to the cluster to distribute the load. This is done by adding replica shards for read-heavy workloads and helps with larger user traffic. We can add load balancing across multiple nodes as well to reduce some bottlenecks. After that, there’s caching which reduces the load on the search system and makes frequent queries come up faster. Then there’s index optimization which improves the overall search performance. That helps the entire system as a whole. Last but not least, I’d check the monitoring. I’d look at where node cpu utilization is too high, where searches are slowing down, which queries are taking the longest to load, then work backwards from the problem. Whenever I look at ways of improving a system, I always think about what’s wrong at the moment. All of that involves probing, learning more, and then building solutions from there. Cloud Level 1: Basic Knowledge 1. What is your favorite cloud provider? Why? AWS because I’ve used it. 2. What is the value of cloud? CapEx → OpEx, scaling, reliability, ease of compliance, access to resource, global reach, economies of scale, access to managed services. Level 2: Knows the Value 1. What is the different between cloud providers, regions, and zones? What do you need to consider for choosing to run your application? The biggest difference is cost and ecosystem. Some larger cloud providers like AWS and Azure have been in the game longer so their ecosystem is much larger. If you have a problem or need access to support, then there’s a good chance someone else has had the same issue. The other consideration is features. Most will have access to a VM, database, and some form of object storage, but the SLAs, history, features, AI, and redundancy attached to the main services differ. Some managed services provide different features than others. Regions are a collection of availability zones where availability zones are a cluster of data centers. It’s important to choose regions closest to customers for lowest latency. 2. What is your preferred way to run a cloud application - IaaS, PaaS, or SaaS? If all choices were available, how would you choose between them? My favorite is IaaS since it gives the most control. IaaS: More control over infrastructure, scalable, create custom apps, manage OS, higher learning curve, EC2/VMs PaaS: Simple app development, underlying infrastructure managed, built in scale, vendor lock in, lacks customizability, AWS Elastic Beanstalk SaaS: Fully managed, no setup or maintenance, instant implementation, less control over data, expensive subscriptions, Salesforce/Slack 3. Why would you choose a multi-cloud strategy? Multi-cloud can be good for DR, fear of vendor lock in, and scale. Level 3: Practitioner 1. Imagine you have a portfolio of 100 on-prem apps that you are responsible for. You are asked to select a list of candidates for a migration to the cloud. How do you make the list? What are the most important factors you need to consider? I’d look at easy things to migrate and test. I’d consider compatibility, the ability to update, integrate AI features, build managed services around it, if it needs reliability, compliance, and security. Something that really tests the limits of cloud and allows the company to see the capabilities and features. 2. If you had to help a customer save 25% on overall cloud spend in the next 12 months, how would you do it? I’d start with the easiest wins that require very little developmental effort. Go away from on-demand instances, turn off unused resources, use cold storage where possible, optimize the backup and DR plans because those typically take up the most resources, resize instance types. Then I’d look at restructuring the application, if necessary, to something like serverless, event-driven workloads that cost less overall but have exponential benefits. Those are harder to implement. Solutions Architect Level 1: Basic Knowledge 1. How have you moved a TB of data from one system to another? For most of my customers, I use tools like AWS DataSync to move data from on-prem to AWS. The company was moving from on-prem because of the recent hurricanes in Florida and they didn’t think their data center was a safe place to house everything. 1. First, I assessed the source and target system compatibilty and looked at their network transfer speed. This customer had good network speed at around 1 Gbps so I could use an online transfer service but if their network was slow, I would’ve recommened something offline like AWS Snowball. The entier transfer was around 5 TB so it’d take about 12 hours. 2. Next, I helped the customer make sure their data was in an understandable file format because if we didn’t do that, then all of the files would’ve randomly been dropped into an S3 bucket. It helps so in the future we do things like data life cycles and add permissions based on files. 3. After that, I helped the customer install the AWS CLI on-prem and monitored the upload with CloudWatch metrics and observed the changes with Alerts/custom dashboard. Level 2: Knows the Value 1. Have you ever paid for software that was open source or free? What convinced you to actually open your wallet? I paid for CloudFlare to get advanced analytics for the dealership I run. I needed the advanced analytics so I could block people from specific IPs from accessing the site. There was a DDoS attack on the contact us form that was a basic JavaScript attack and getting their Page Shield service was a must. It also allowed me to get alerts so I could see when it was happening or if it was something with ISP/GoDaddy being slow. Level 3: Practitioner 1. It’s black friday and you need to prepare to 10x your architecture. What are some strategies you could use to scale it? Prescale based on predetermined metrics based on which service will experience however many requests. Similar to Netflix. Netflix operates at full active across 4 AWS Regions. Netflix’s biggest issue when dealing with these increases is the Mean Time to Detection. The proper metrics need to be in place to scale up because it doesn’t matter how fast resources can spin up, if the scaling is detected too late then it’ll slow down. ○ This requires higher resolution metrics. Going from 5min resolution → 1min resolution. Netflix even went down to 5sec resolution. ○ You’re only as fast as your slowest system.  ○ Do parallel startups for separate services rather than sequential then join them. Horizontal scaling with auto-scaling groups + load balancing Use Caching Add read replicas fo the database with managed scaling Build out IaC to scale the infrastructure or duplicate it in another region for scale/DR Build real-time monitoring dashboards with alerts for things like spikes in latency, cpu, database usage. Use rate limiting to prevent bots from scraping product pages and IP blacklists. Real-time security monitoring with anomaly detection and have a SOC connected to a SIEM. Build a shared criticality nomenclature. Define tags and figure out what is truly mission critical. In Netflix’s situation, it’s more important to allow users to start a show and not see recomendations when there’s degradation. A customer had the exact same scenario because they held a big sale for their parks and rec website. We had to do that. 2. You are assigned a task to move 10 apps of varying sizes from one data center to another. How?