Impala JDBC connection with Kerberos
The idea was to use Java locally (in my case with InteliJ) to connect to Hive metastore through Impala. That was in order to read some data and then be able to use them by some other processes on later stages. Hadoop cluster that I was connecting to was Kerberised, which made the exercise more tricky. Here’s how I managed to establish such a connection.
HOW TO
I was using InteliJ Community Edition 2019.2.3 and cluster with CDH 6.1.1 with Kerberos.
In order to connect to the Hadoop cluster, at first I had to obtain a Kerberos ticket. I used kinit for that:
|
1 |
kinit my_user@my_domain |
After providing the password, I checked that my ticket was created:
|
1 |
klist |
You should be seeing something like this:
|
1 2 3 |
Credentials cache: API:123:4 Principal: my_user@my_domain Cache version: 0 |
You may also obtain your ticket from the keytab in order not to manually type password each time:
|
1 |
kinit my_user@my_domain -k -t my_keytab |
Additionally I needed the krb5.conf file, which I downloaded from the cluster. It was in /etc/krb5.conf location in this case.
Then I had to get the Impala JDBC driver. I found it here.
With those pieces I was able to connect and list/access all the tables available in my metastore.
Here’s my code:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
import org.apache.hadoop.security.UserGroupInformation; import java.io.IOException; import java.sql.*; public class ImpalaJDBC { public static void main(String[] args) throws IOException { // Attaching Kerberos conf information System.setProperty("java.security.krb5.conf", "krb5.conf"); System.setProperty("sun.security.krb5.debug", "true"); try { // Obtaining Hadoop conf for connection org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration(); conf.set("hadoop.security.authentication", "kerberos"); UserGroupInformation.setConfiguration(conf); // Specifying Impala driver Class.forName("com.cloudera.impala.jdbc4.Driver"); // Establishing connection System.out.println("getting connection"); Connection con = DriverManager.getConnection("jdbc:impala://my_impala_server:impala_port/ my_db;AuthMech=1;KrbHostFQDN=my_impala_server;KrbServiceName=impala"); System.out.println("got connection"); // Checking if connection works PreparedStatement statement = con.prepareStatement("show tables"); ResultSet rs = statement.executeQuery(); int columns = rsmd.getColumnCount(); while (rs.next()) { for (int i = 1; i <= columns; i++) { if (i > 1) System.out.print(", "); String value = rs.getString(i); System.out.print(value); } System.out.println(""); } con.close(); } catch (SQLException | ClassNotFoundException e) { e.printStackTrace(); } } } |
In pom.xml I only needed to attach hive-jdbc.jar. This is how it looked like:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelversion>4.0.0</modelversion> <groupid>com.cloudera.hive</groupid> <artifactid>impala-jdbc</artifactid> <version>1.0</version> <packaging>jar</packaging> <name>impala-jdbc</name> <properties> <project.build.sourceencoding>UTF-8</project.build.sourceencoding> </properties> <dependencies> <dependency> <groupid>org.apache.hive</groupid> <artifactid>hive-jdbc</artifactid> <version>3.1.2</version> </dependency> </dependencies> <build> <plugins> <plugin> <artifactid>maven-assembly-plugin</artifactid> <version>2.4</version> <configuration> <descriptorrefs> <descriptorref>jar-with-dependencies</descriptorref> </descriptorrefs> </configuration> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> <plugin> <groupid>org.apache.maven.plugins</groupid> <artifactid>maven-compiler-plugin</artifactid> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> </plugins> </build> </project> |
And that was it. You can apply more complex queries to your tables or redirect the data to some other place.