Dataiku Core Designer: Interface & Data Exploration

Projects

• Projects are central workspaces in Dataiku, hosting data, recipes, models, discussions, and dashboards related to specific tasks.
• Created from the homepage, projects can be organized into folders for better management.
• Users can monitor project status, review recent activity, view contributors, and manage to-do lists within a project.
• Key operations on projects include duplication, exporting, and deletion, depending on access levels.

Flow

• The Flow serves as a visual pipeline showcasing how data, recipes, and models interact for analysis.
• It maps the data journey from source to output highlighting dependencies among components.
• The Flow layout is automatically optimized and cannot be manually adjusted.

Enhancing Flow Readability

• Use Flow Zones to group related items in the Flow for improved clarity.
• Add tags to Flow items for filtering by attributes such as creator, purpose, or status.
• Use the View menu to apply filters to the Flow based on zones, tags, connections, recipe engines, etc.

Building the Flow

• Option to build the entire Flow with the "Build all" function, or individually by right-clicking items and selecting "Build."
• Changes to datasets or recipes trigger dynamic rebuilding of dependent items upstream or downstream in the Flow.

Datasets

• Datasets in Dataiku represent data stored and processed in a tabular format.
• Examples include uploaded Excel spreadsheets, SQL tables, data files on Hadoop clusters, or cloud-based CSV files.
• Datasets are visually represented in the Flow with blue squares and icons specific to their source type.
• The interface for interacting with datasets is consistent regardless of the source, providing tabs for Explore, Charts, Statistics, Data Quality, Metrics, History, and Settings.
• Dataiku separates processing logic from storage infrastructure allowing for uniform interaction across various dataset types.
• Instead of ingesting whole datasets, Dataiku stores connection details and retrieves data from the source, only transferring a sample when necessary.

Data Connections

• Dataiku allows manipulation of externally stored datasets through connections to sources such as SQL databases, cloud storage, and NoSQL sources.
• Importing new datasets into the Flow can be achieved by uploading files or leveraging existing connections.
• The user interface for exploring and preparing data remains consistent across different dataset types due to decoupling of processing logic from the storage infrastructure.

Managing Connections

• Instance administrators control connection configurations through the Connections menu in Dataiku Cloud.
• In self-managed installations, administrators manage connections in Applications > Administration > Connections menu.
• Administrators control credentials, security settings, usage parameters and can add new connections.
• Plugins provide additional connection types, separating data connection management from data usage for streamlined workflows.

Exploring Data

• Columns in Dataiku datasets represent the features of the data.
• Each column has two main attributes: storage type and meaning.

Storage Type

• Defines how data is stored in Dataiku influencing applicable transformations.
• Includes types like String, Integer, Float, Boolean, and Date.
• It is important to avoid altering the storage type for datasets imported from external sources like SQL tables.

Meaning

• Semantic label indicating the nature of the column, such as country, email, or temperature.
• Dataiku infers meanings from column values, but they can be adjusted for greater accuracy or customized.
• Meanings aid in auto-detecting possible transformations, measuring data quality, and simplifying value searches.

Dataset Schema

• Lists column names and storage types associated with a dataset.
• Viewable and editable in the Schema tab of the right panel either during dataset upload or later in the workflow.

Sampling Benefits

• To manage computational load and provide immediate visual feedback, Dataiku displays only a sample of large datasets.
• Used during visualization, data preparation, and statistical analysis to facilitate quick actions such as sorting, filtering, displaying column distributions, applying conditional formatting, and viewing summary statistics on a smaller sample.

Sampling Methods

• By default, Dataiku samples the first 10,000 rows. However, this can introduce bias.
• To address bias, users can adjust the sampling method, choosing from options like random, stratified, or class rebalancing through the Sampling settings panel.
• While more representative samples provide better insights, they may require more processing time.

Analyze Window

• Enables in-depth investigation of column values in the Explore tab.
• Accessible via the context menu of a column header.
• Provides insights into the data sample's quality and distribution.

Analyze Window Overview

• Navigation: Use arrows at the top left to switch between adjacent columns and view metrics for other columns.
• Sample Management: Analysis defaults to the dataset sample, but can be adjusted to calculate for the entire dataset.

Analyze Window Tab Descriptions

• Categorical: Displays a bar chart of the most frequent observations.
• Numerical: Presents a histogram, boxplot, summary statistics, frequent values, and outliers.
• Values Clustering: Groups similar values to standardize text fields with unwanted variations.
• Both tab types include a Summary section reviewing data quality, showcasing valid, invalid, empty, and unique values.

Charts

• The Charts tab provides a drag-and-drop interface for visualizing data, offering various chart types such as bar charts, line graphs, pivot tables, and scatter plots.

Building a Chart

• Select a chart type and drag variables from the Data panel to the desired axis to create a chart.

Customizing a Chart

• Configure chart variables by selecting the appropriate aggregation, adjusting value sorting, and modifying value and label formatting.
• Additional features include zooming on time series, changing date intervals, and exploring multiple series side-by-side.

Filtering Data

• Filter data by selecting specific groups or combining less-prevalent categories into an "Others" bucket.
• Filters can be applied directly from a chart tooltip.