Q01 What is Spark streaming? A01 Spark streaming extends Spark core/sql to what is called a micro-batching to give Near-Real-Time processing where the arriving live stream of data from sources like Kafka, Kinesis, Flume, HDFS file system, etc divided into batches of predefined interval. … Read more ›...
Q1. What is AWS Athena? What are its capabilities? A1. Amazon Athena is an interactive query service that makes it easy to analyse data in Amazon S3 buckets using standard SQL. Athena is serverless, … Read more ›...
Q1. What is a Slowly Changing Dimension (i.e. SCD)? A1. SCD means the dimensions that change slowly over time, rather than changing on regular basis. For example, change in customer name or address. There are different types of changing dimensions, and type 1 & … Read more ›...
Q1. What are the different Spark SQL join types?
A1. There are different SQL join types like inner join, left/right outer joins, full outer join, left semi-join, left anti-join and self-join.
Q2. Given the below tables, can give examples of the above join types?
…
Over the last few years the number of AWS services that are related to building, managing or deploying software projects on AWS has grown a lot — AWS Amplify, AWS App Runner, AWS Elastic Beanstalk, AWS Code family like CodePipeline, CodeCommit, CodeBuild, CodeStar, CodeArtifact, and Amazon CodeCatalyst. This post does...
What is a data pipeline in cloud? A data pipeline is a means of moving data from one place (E.g source systems) to another (E.g. target/destination). The data is transformed and optimized along the way so that it can be analyzed and used to develop business insights. … Read more...
As the demand for containerised applications continues to grow, expertise in AWS ECS has become increasingly valuable for software & data engineers. Q01. What are the benefits of container orchestration services? A02. Container orchestration automates container deployment, management, scaling, and networking. Orchestration tools manage these complexities, … Read more ›...
Below questions are based on the above diagram, and extends AWS Q&As on VPC, Subnets, Availability Zones, VPN, Route tables, NACLs & Security Groups & 15+ AWS Services overview interview Q&As. Q1. What is the purpose WAF in AWS? … Read more ›...
Q01. What is the difference between Azure Data Factory and AWS Glue? A01. Azure Data Factory and AWS Glue are two competing serverless PaaS (i.e. Platform as a Service) based ETL and ELT solutions from two major cloud providers Azure and AWS respectively. … Read more ›...
Q01. What are the key pillars of Data Governance? A01. The 5 key pillars of any Data Governance are: 1) Security & privacy – Who can access it? Personally identifiable information (i.e. PII) (i.e. any data that could potentially identify a specific individual? … Read more ›...
Q01. What is an idempotent data operation? A01. Idempotent operations produce the same result even when the operation is repeated many times. A pipeline that reads data from a number of source systems and loads it into target RDBMs tables more than once for a given day can result in...
Q01. What do you understand by the term canonicalizing? A01. Canonicalizing is an activity of replacing multiple copies of an object/entity/URL with just a few objects/entities/URLs. A canonical URL is a good example. If you have a single page that’s accessible by multiple URLs, … Read more ›...
Common Table Expressions or CTE’s for short are used in SQL to simplify complex joins and subqueries, and to provide a means to query hierarchical data such as an organisational chart. This was also introduced in MySQL 8 – Using https://www.db-fiddle.com/. … Read more ›...
I have a good news, I got job offer. I like to share with you from my bottom of my heart feeling thankful to you.