2021

AWS Big Data IOT With Spark StreamingBusiness case :Use AWS Big data services to process streaming data from IOT device.iotawsspark

2020

Airflow Pipeline Kafka CassandraBusiness case : Create Airflow Pipeline using Kafka for Realtime stock price streaming, Cassandra for Realtime stock data warehousing Real time stock price can be streamed from alphavantage.com(subscription required)....airflowpipelinedataMQTT Spark Streaming With Watermark Part 2Earlier we created a IOT Device / MQTT Client using Raspberry Pi.iotmqttsparkMQTT Publish Subscribe IOT Part 1We will create a IOT Device / MQTT Client using Raspberry Pi.iotmqttsparkCustom MQTT Spark ReceiverSpark Streaming’s Receivers accept data in parallel and buffer it in the memory of Spark’s workers nodes.iotawssparkSpark SQL HiveIn this post we will explore Spark SQL reading processing and writing data stored in Apache Hive.sparkawshiveSpark Structured Streaming Integrate Kafka Streams to Cassandra SinkKafka Streams integrated with Cassandra Sink using Spark Structured Streaming.sparkkafkacassandraKafka Streamskafka Stream Word Count.kafkastreaming datastatelessInject Spark Stream to ElastiSearch Sink With KibanaElastisearch is a distributed document search and analytics engine, real-time search.sparkkafkaelastisearchAWS EMR SparkSteps for Deploying Spark App on Amazon EMR Make sure there are no paths to your local filesystem used in your script!sparkawsemrCassandraSetup Multi-node Cassandra cluster on a single machine We will create three node Cassandra cluster on single local machine.cassandrascalajekylHue Oozie HdfsHue provides UI for main hadoop features.ooziehuehdfsFlink StreamingFlink is a stream processing engine.flinkstreamingscalaPresto Hive CassandraPresto can connect to many different “big data” databases and data stores at once, and query across them using SQL syntax.sparkawsemrSpark PostGres Cassandra MongoDbIntegrate spark with Postgres, Cassandra,MongoDb Explore how to write spark streaming data into databases.sparkpostgrescassandraTwitter Custom Spark ReceiverCreate your twitter feed filtering the handles you follow and also filtering out retweets by those handles using custom Spark receiver.sparkstreamingreceiverSpark PartitioningSpark isn’t totally magic – you need to think about how your data is partitioned Operations that are expensive, and Spark won’t distribute data on its own.sparkscala

2019

Kafka Cluster- Kafka is simply a collection of topics split into one or more partitions.kafkaclustermongodbUnderstanding Consensus algorithms in Distributed SystemsConsensus Algorithms Consensus algorithms allow a collection of machines to work as a coherent group that can survive the failures of some of its members.raftelectionclusterKafka Zookeeper Cluster Leader ElectionThis article will configure kafka zookeeper cluster and explore the leader election process.kubernetesdockerblockchainKubernetes DockerThe objective of this post is to implement a basic hyperledger Block Chain network on Kubernetes platform.kubernetesdockerblockchain

2015

Technical BloggingHow I started.bloggingbigdata