WORKING FILES
〰️
WORKING FILES 〰️
Apache Spark Training
〰️
Apache Spark Training 〰️
Step 1: Local Laptop Cluster Setup
Teach them Local laptops Cluster Setup… then ask them to do it on another laptop and try
Anaconda3-2022.10-Windows-x86_64
https://www.apache.org/dyn/closer.lua/spark/spark-3.3.1/spark-3.3.1-bin-hadoop2.tgz
https://github.com/steveloughran/winutils/archive/refs/heads/master.zip
Environment Variables (windows)
HADOOP_HOME = C:\SPARK\hadoopJAVA_HOME = C:\Program Files\Java\jdk1.8.0_202SCALA_HOME = C:\SPARK\scalaSPARK_HOME = C:\SPARK\sparkPYSPARK_PYTHON = C:\Users\user\anaconda3\python.exePATH Variables:%SPARK_HOME%\bin%HADOOP_HOME%\bin%SCALA_HOME%\bin%JAVA_HOME%\bin
Jupyter Environment Variables
PYSPARK_DRIVER_PYTHON = C:\Users\User\anaconda3\Scripts\jupyter.exePYSPARK_DRIVER_PYTHON_OPTS = notebookSTep 2: Local Spark Shell
Step 3: Spark cluster in Google Cloud
Step 4: Spark cluster in AWs Databricks
Step 5: Spark on colab
How To Start A Spark Session & Read in CSV frrom Website.ipynb
Datacamp Pyspark Cheatsheets
Step 6: Pyspark DataFrames
Dataframes with PySpark Example 1 by Dr Alvin.ipynb
Dataframes with PySpark Example 2 by Dr Alvin.ipynb
purchases.csv (this dataset has 4 millions rows.. u can’t open in Excel!)
DataFrames with PySpark Example 2 (EXERCISE) by Dr Alvin Ang.ipynb
Step 7: Showcase VAEX
VAEX for Crunching Big Data by Dr Alvin Ang.ipynb
Automated_Traffic_Volume_Counts.csv (this dataset is 3GB!!!)
Others
hierarchical-clustering-with-python-and-scikit-learn-shopping-data.csv
Structured_Streaming_with_PySpark.ipynb
KMEANS_PySpark_by_Dr_Alvin.ipynb
Hierarchical_Clustering_In_Spark_With_Bisecting_K_Means.ipynb
Normalizer,_Scaler,_Bucketizer_and_Binarizer_with_PySpark_by_Dr_Alvin_Ang.ipynb
https://www.linkedin.com/in/nandeshreddy/
Soya Kim - Big Data Healthcare
Spark Commands by Raj Bharat.txt
https://www.youtube.com/watch?v=lisIQ9ohU8g
https://www.datanami.com/2024/03/05/duckdb-walks-to-the-beat-of-its-own-analytics-drum/
Python Training
〰️
Python Training 〰️
List / Tuples / Dictionary / Sets
Functions
OOP
Intermediate
[Comprehension.i.for.i.in.range] + [Mounting.Google.Drive.File.IO] + [Object.Oriented.Programming.Class.Inheritance .Parents] + [SQLite.Databasing] + [Error.Handling.IndexError.IOError.TypeError.ValueError]Mounting and Creating Files in Google Drive with Colab.ipynb
Updating HR Records using Object Oriented Programming by Dr. Alvin Ang
Python PDFS
30 Python Libraries to Boost Your Data Science Productivity.pdf
Python Data Science Tips Full Archive by avi chawla. subs tack. com
https://www.hospitalmanagementasia.com/tech-innovation/harnessing-the-power-of-data-in-healthcare/ (Sean Singhealth)
https://developers.googleblog.com/en/data-science-agent-in-colab-with-gemini/
Python Cheatsheets
PANDAS Training
Loops
Statistics
[Descriptive.Stats] + [Seaborn.Visualization] + [Hypothesis.Testing.ANOVA] + [LR.MR.R2]Day 4 with Dr Alvin.ipynb (Statistics)
Python Visualization
Python Specials
NUMPY
Django and Flask
File IO
Exception / Error Handling
RegEx
Data Cleansing and Wrangling
〰️
Data Cleansing and Wrangling 〰️
Steps in Data Wrangling and Cleansing
Converting JSON to CSV from Scikit Learn Datasets by Dr. Alvin Ang.ipynb
How to Do Train Test Splits with Python by Dr Alvin Ang.ipynb
Data Wrangling a Population of Countries’ Dataset
Population of Countries in 2000.csv
Data Wrangling a Population of Countries Dataset by Dr Alvin Ang.ipynb
Data Wrangling & Visualizing Healthcare Datasets
Hospital Admissions
hospital-admissions-by-sector-annual.csv
Data Cleansing a Hospital Admissions Dataset by Dr Alvin Ang.ipynb
Health Expenditure
government-health-expenditure.csv
Data Cleansing a Government Health Expenditure Dataset.ipynb
Long Term Care Facilities
number-of-residential-long-term-care-facilities-sector-breakdown.csv
Data Wrangling a Long Term Care Facilities Dataset by Dr Alvin Ang.ipynb
Data Cleansing a Rock Song Dataset
Data Cleansing a Rock Song Dataset with Python by Dr. Alvin Ang.ipynb
Data Wrangling Air Quality Datasets
Data Wrangling Air Quality Datasets with Python by Dr Alvin Ang.ipynb
Searching and Slicing a Video Games Dataset
Searching and Slicing a Video Games Dataset with Python by Dr Alvin Ang.ipynb
Wrangling Automobile Datasets
Slicing & Dicing a Motorcars Dataset (European + Japanese Cars)
Slicing & Dicing a Motorcars Dataset with Python by Dr Alvin Ang.ipynb
Factors Affecting Price of European and Japanese Cars
Feature Selection on Automobile Dataset with Python by Dr. Alvin Ang.ipynb
Dealing with Missing Data in European Cars Dataset
Data Cleansing a European Automobile Dataset with Python by Dr Alvin Ang.ipynb
Cross Validating a European and Japanese Cars Dataset
Cross Validating a European and Japanese Car Dataset by Dr Alvin Ang.ipynb
Python for Finance
https://www.kaggle.com/datasets/wordsforthewise/lending-club
https://drive.google.com/file/d/1VCaoIFxzpYgzCaIerj24EWx-wW87C440/view?usp=drive_link (alvin’s google drive FULL LendingClubLoan Dataset)
Feature Selection on Lending Club Loan Dataset by Dr Alvin Ang.ipynb
Data Cleansing the Lending Club Loan Dataset by Dr Alvin Ang.ipynb
Train Test Splitting the Lending Club Loan Dataset by Dr Alvin Ang.ipynb
https://wmi.edu.sg/dmp-online-singapore/ RAY DALIO
Hypothesis Testing and ANOVA with Python
Hypothesis Testing and ANOVA with Python by Dr Alvin Ang.ipynb
R Training
〰️
R Training 〰️
Datacamp R Cheatsheets
R Sites
https://www.r-bloggers.com/2022/06/the-most-overlooked-r-package-that-can-get-you-through-a-data-science-job-interview/
https://online.stat.psu.edu/statprogram/tutorials/statistical-software/r
https://cran.r-project.org/web/packages/available_packages_by_name.html
https://www.kaggle.com/code/rtatman/data-cleaning-challenge-json-txt-and-xls
https://www.business-science.io/finance/2020/02/26/r-for-excel-users.html
Text Mining with R
Statistics with R
Tableau Training
〰️
Tableau Training 〰️
1st Project
Extras
Different Map Types in Tableau
2nd Project
Storytelling
Change Over Time:
https://public.tableau.com/app/profile/ben.jones/viz/WorldPopulationDay/1_ChangeOverTime
https://public.tableau.com/app/profile/andy.kriebel/viz/EPLInjuries/InjuryCrisis
Drill Down:
https://public.tableau.com/views/EarthquakesOnTheRise-Full/Earthquakestory
https://public.tableau.com/app/profile/mac.bryla/viz/TellmeaboutWill/TellmeaboutWill
Zoom Out:
https://public.tableau.com/app/profile/halftimeheroes/viz/OlympicGamesStories-ZoomOut/ZoomOut
Contrast:
https://public.tableau.com/app/profile/robertrouse/viz/Pyramids_1/EgyptianPyramids
Outliers
Tableau Desktop Specialist
Datacamp Tableau Cheatsheets
Tableau Data Analyst
SQL Training
〰️
SQL Training 〰️
Datacamp SQL Cheatsheets
For those with problems installing….
Oracle SQL
〰️
Power BI Training
〰️ Power BI Training
Datacamp Power BI Cheatsheets
Resource Allocation with Excel Solver
〰️
Resource Allocation with Excel Solver 〰️
Statistics Training
〰️
Statistics Training 〰️
Degree of Freedom
Statistics Cheatsheets
15 Data Fallacies
Excel Training
〰️
Excel Training 〰️
Data Analytics with Excel Course
Power Query / Power Pivot
〰️
Power Query / Power Pivot 〰️
Dashboard with Excel
〰️
Dashboard with Excel 〰️
Design of Experiments (DOE)
〰️
Design of Experiments (DOE) 〰️
Flexsim
〰️
Flexsim 〰️
WEKA
〰️
WEKA 〰️
GOOGLE WORKSPACE
〰️
GOOGLE WORKSPACE 〰️
Google Sheets
〰️
Google Sheets 〰️
Looker Studio
〰️
Looker Studio 〰️
slides for Looker Studio (Part 2 - Theme and Formatting).pdf
slides for Looker Studio (Part 3 - Agg Functions and Table Calculations.pdf)
slides for Looker Studio (Part 4 - Drill Down and Pivot Tables.pdf)
slides for Looker Studio (Part 5 - Calculated Fields and Filters.pdf)
slides for Looker Studio (Part 6 - Case When & Blending Data.pdf)
Tech with Tim's Facial Recognition Project (on local laptop)
〰️
Tech with Tim's Facial Recognition Project (on local laptop) 〰️
Data Quality Training
〰️
Data Quality Training 〰️
Bodies
Open Refine
https://datacarpentry.github.io/OpenRefine-ecology-lesson/aio.html
https://web.archive.org/web/20190105063215/http://enipedia.tudelft.nl/wiki/OpenRefine_Tutorial
Tools
Datasets
Blogs
https://atlan.com/data-governance-framework/?ref=/open-source-data-governance-tools/
https://atlan.com/master-data-management-vs-metadata-management/
https://www.gartner.com/smarterwithgartner/how-to-improve-your-data-quality
https://tdan.com/category/data-topics/data-quality-articles-blogs-education
Singapore Public Service
Slides and Whitepapers
EY - Becoming an Analytics Organization.pdf
Creating an Enterprise Data Strategy - Beye Network.pdf
A Definitive Guide to Data Governance - Trillium Software.pdf
TADA Data Quality Concepts.pdf
Introduction to Data Governance and Stewardship - Salesforce.pdf
5 Levels of Master Data Management Maturity - Baseline Consulting.pdf
Dataprep - Acclerate Data for AI.pdf
ISO Data Quality 8000-1 (partial).pdf
ISO-8000-61-2016 (partial).pdf
A Product Perspective on Total Data Quality Management - Richard Wang.pdf
Datacamp Data Quality Cheatsheets
Datacamp - Cheat Sheet DS For Business Leaders.pdf
Datacamp - Data Quality Dimensions.pdf