Managing Big Data in Clusters and Cloud Storage
This comprehensive course on managing big datasets teaches learners how to load data into clusters and cloud storage and how to structure it for queries using distributed SQL engines like Apache Hive and Apache Impala. It covers tools to browse databases and explore files, as well as how to create and manage databases and tables. Learners will also learn to choose appropriate data types, storage systems, and file formats. The course is suitable for those interested in data management and those seeking to gain essential skills for navigating today's data-driven world.
per person
Level
Duration
Training Delivery Format
Face-to-face / Virtual Class
per person
Level
Duration
Training Delivery Format
Face-to-face (F2F) / Virtual Class
Class types
Public Class
Private Class
In-House Training
Bespoke
About this course
n this course, you’ll learn how to manage big datasets, how to load them into clusters and cloud storage, and how to apply structure to the data so that you can run queries on it using distributed SQL engines like Apache Hive and Apache Impala. You’ll learn how to choose the right data types, storage systems, and file formats based on which tools you’ll use and what performance you need.
What you will learn:
-
Use different tools to browse existing databases and tables in big data systems
-
Use different tools to explore files in distributed big data filesystems and cloud storage
-
Create and manage big data databases and tables using Apache Hive and Apache Impala
-
Describe and choose among different data types and file formats for big data systems
Who should attend?
Ideal for professionals who are working with large datasets, such as data analysts, data engineers, and database administrators. It is also suitable for individuals who want to develop their skills in managing big data and storing it in distributed systems like clusters and cloud storage. Prior experience with SQL, databases, and cloud computing is recommended but not required, making it accessible to a wide range of learners. Ultimately, anyone looking to gain a deeper understanding of big data management and its applications would benefit from attending this training.
Learning Outcome
By the end of the course, you will be able to
• use different tools to browse existing databases and tables in big data systems;
• use different tools to explore files in distributed big data filesystems and cloud storage;
• create and manage big data databases and tables using Apache Hive and Apache Impala;
• describe and choose among different data types and file formats for big data systems.
Prerequisites
To use the hands-on environment for this course, you need to download and install a virtual machine and the software on which to run it. Before continuing, be sure that you have access to a computer that meets the following hardware and software requirements:
• Windows, macOS, or Linux operating system (iPads and Android tablets will not work)
• 64-bit operating system (32-bit operating systems will not work)
• 8 GB RAM or more
• 25GB free disk space or more
• Intel VT-x or AMD-V virtualization support enabled (on Mac computers with Intel processors, this is always enabled; on Windows and Linux computers, you might need to enable it in the BIOS)
• For Windows XP computers only: You must have an unzip utility such as 7-Zip or WinZip installed (Windows XP’s built-in unzip utility will not work)
Course Content
Module 1: Orientation to Data in Cluster and Cloud Storage
- Browsing Tables with Hue
- Browsing Tables with SQL Utility Statements
- Browsing HDFS with the Hue File Browser
- Browsing HDFS from the Command Line
- Understanding S3 and Other Cloud Storage Platforms
- Browsing S3 Buckets from the Command Line
Module 2: Defining Databases, Tables and Columns
- Introduction to the CREATE TABLE Statement
- Using Different Schemas on the Same Data
- Specifying TBLPROPERTIES
- Examining, Modifying, and Removing Tables
- Hive and Impala Interoperability
- Impala Metadata Refresh
Module 3: Data Types and Files Types
- Overview of Data Types
- Choosing the Right Data Types
- Overview of File Types
- Choosing the Right File Types
Module 4: Managing Datasets in Cluster and Cloud Storage
- Refresh Impala’s Metadata Cache after Loading Data
- Loading Files into HDFS with Hue’s Table Browser
- Loading Files into HDFS with Hue’s File Browser
- Loading Files into HDFS from the Command Line
- Loading Files into S3 from the Command Line
- Using Hive and Impala to Load Data into Tables
- Conclusion
Module 5: Optimizing Hive and Impala (Honors)
- What to Do When Queries Are Too Complex
- What to Do When Queries Take Too Long
- When to Use Table Partitioning
- When to Use Complex Columns
- File Systems versus Storage Engines
At this time, this course is available for private class and in-house training only. Please contact us for any inquiries.