xgboost, or Extreme Gradient Boosting is a very convenient algorithm that can be used to solve regression and classification problems. You can check may previous post to learn more about it. It turns out we can also benefit from xgboost while doing time series predictions.
Read moreThe idea was to use Java locally (in my case with InteliJ) to connect to Hive metastore through Impala. That was in order to read some data and then be able to use them by some other processes on later stages. Hadoop cluster that I was connecting to was Kerberised, which made the exercise more tricky. Here’s how I managed to establish such a connection.
Read moreSpark AI Summit Europe, which happened in October, was full of interesting stuff. It was mostly focused on features coming with Spark 3, but not only. Preview release of Spark 3 is already available and can be obtained here. There are a lot of cool features planned, especially when it comes to making Data Science easier on big data and with Spark in particular.
Read moreHow to put AI projects in production – that’s a topic that since a while occupies heads of many data scientists and plays a key role on many international conferences. Unfortunately there is no one clear answer to that, as the variety of topics that we nowadays name as AI is huge.
Read moreData forecasting is a process of estimating the future based on historical values. It is described by time series, which is simply a series of time dependent data points. We usually forecast different costs or sales over time. We can try to predict weather conditions or model stock changes. Basically look at any process that can be described as time dependent with certain time interval (hourly, daily, monthly…).
Read moreR has a nice package support when it comes to forecasting. There is the forecast package which allows you to build several types of models
Read moreMany times there is a demand from end users to deliver results in a form of Excel files. The bigger and more complex the files grow, the more difficult it is to properly interpret them. This is when nice formatting can help you out.
Read moreXGBoost or in long version Extreme Gradient Boosting got recently very popular, especially on Kaggle competitions. It proved to outperform many other algorithms on tasks such as classification and regression. I used it few times as well and that’s why I decided to take a closer look into XGBoost to see how it works.
Read moreWith Apache Beam we can connect to different databases – HBase, Cassandra, MongoDB using specific Beam APIs. We also have a JdbcIO for JDBC connections. Here I show how to connect with MSSQL database using Beam and do some data importing and exporting in Kerberised environment.
Read moreHBase is a NoSql database, which allows you to store data in many different formats (like pictures, pdfs, textfiles and others) and gives the ability to do fast and efficient data lookups. HBase has two APIs to chose from – Java API and HBase Shell. We can also connect HBase with some different tools like Hive or Phoenix and use SQL. HBase also integrates with Apache Beam via HBaseIO transform.
Read more